HEX
Server: LiteSpeed
System: Linux s787.bom1.mysecurecloudhost.com 4.18.0-477.13.1.lve.el8.x86_64 #1 SMP Thu Jun 1 16:40:47 EDT 2023 x86_64
User: mobilech (5348)
PHP: 8.1.34
Disabled: NONE
$ pwd: /opt/saltstack/salt/lib/python3.10/site-packages/salt/__pycache__/
Upload Files
File: //opt/saltstack/salt/lib/python3.10/site-packages/salt/__pycache__/minion.cpython-310.pyc
o

;j4A�@s(dZddlZddlZddlZddlZddlZddlZddlZddlZddl	Z	ddl
Z
ddlZddlZddl
Z
ddlZddlZddlZddlZddlZddlZddlmZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Zddl!Zddl"Zddl#Zddl$Zddl%Zddl&Zddl'Zddl(Zddl)Zddl*Zddl+Zddl,Zddl-Zddl.Zddl/Zddl0Zddl1Zddl2Zddl3Zddl4Zddl5Zddl6Zddl7Zddl8Zddl9Zddl:Zddl;Zddl<Zddl=m>Z>ddl?m@Z@ddlAmBZBddlCmDZDmEZEmFZFmGZGmHZHmIZImJZJmKZKmLZLddlMmNZNdd	lOmPZPdd
lQmRZRddl/mSZSddl5mTZTdd
l7mUZUmVZVmWZWddl<mXZXmYZYzddlZZZdZ[Wne\�y`dZ[Ynwzddl]Z]dZ^Wne\�yudZ^Ynwzddl_ZdZ`Wne\�y�dZ`Ynwe�aeb�Zce�d�Zedd�Zfd<dd�Zgd=dd�Zhdd�Zidd�Zjd>dd�Zkdd�Zld?d d!�Zmd"d#�ZnGd$d%�d%�ZoGd&d'�d'eo�ZpGd(d)�d)�ZqGd*d+�d+eo�ZrGd,d-�d-eo�ZsGd.d/�d/es�ZtGd0d1�d1eo�ZuGd2d3�d3er�Zvd4d5�Zwd6d7�ZxGd8d9�d9es�ZyGd:d;�d;ep�ZzdS)@z
Routines to set up a minion
�N)�OrderedDict)�	ipaddress)�DEFAULT_MINION_OPTS)�DEFAULT_TARGET_DELIM)	�CommandExecutionError�CommandNotFoundError�SaltClientError�SaltDaemonNotRunning�
SaltException�SaltInvocationError�SaltMasterUnresolvableError�SaltReqTimeoutError�SaltSystemExit)�SLS_ENCODING)�RequestContext)�enable_sigusr1_handler)�tagify)�parse_host_port)�ProcessManager�SignalHandlingProcess�default_signals)�ZMQ_VERSION_INFO�zmqTFcCst��dS)z�
    Signal any in-progress resolve_dns() retry loop to abort on its next
    wakeup. Used by the minion shutdown path so SIGTERM is not blocked by
    a synchronous ``time.sleep`` inside the DNS retry loop.
    N)�_RESOLVE_DNS_ABORT�set�rr�?/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py�request_resolve_dns_abortxsr��?cCsX|dkr|��St��|}	|��rdS|t��}|dkr#|��St�t||��q)a�
    Sleep up to ``duration`` seconds in ``chunk``-second slices, returning
    early if ``abort_event`` becomes set. Returns True if the event was
    observed set (i.e. the sleep was aborted), False otherwise.

    Using small chunks rather than ``abort_event.wait(duration)`` keeps
    behavior consistent across platforms where ``Event.wait`` may starve
    other threads sharing the GIL during very long timeouts.
    rT)�is_set�time�	monotonic�sleep�min)ZdurationZabort_event�chunkZdeadline�	remainingrrr�_interruptible_sleep�s
�r&cCsxi}d}|�dd�dkr|�dd�sd}ddl}|dur�z|d	d
kr%t�|jj�|d	t|d�d|d�|d
<Wn�ty�|�dd�}|dr�	|durZ|dkrVt�|d8}t	�
d|d	|d�t|dt�}|rxt	�
d|d	�t�z|jj�|d	t|d�d|d�|d
<Wn
ty�YnwqLn|r�d|d
<n�Yn6ty�d}|�d	|�}|d
kr�|}|�d�dkr�d�|�}	nd�|�}	t	�
|	�td|	d��wd|d
<d
|vr�d
|vr�|d
|d
kr�t	�
d|d
|d
�|d�rZt	�d|d�|jj��}
t	�d�t	�|
�|d|
v�rQ|
|dd�rH|d�s/|
|dd n|
|dd!}|dd"|d#<t	�d$|d#�n3t	�
d%|d�n*t	�
d&|d�n!|d'�r{|jj�|d't|d(�d|d�|d#<t	�d$|d#�|d(�r�t|d(�|d(<t	�d)|d(�|d*�r�t|d*�|d*<t	�d+|d*�d,j|d
|dd-�|d.<t	�d/|d.�|S)0z7
    Resolves the master_ip and master_uri options
    T�file_client�remote�local�use_master_when_localFrN�master��master_portZipv6Z	master_ip�retry_dns_count�	retry_dns�zIMaster hostname: '%s' not found or not responsive. Retrying in %s secondszPMaster DNS retry loop aborted by shutdown request before '%s' could be resolved.z	127.0.0.1zunknown address�__role�syndiczyMaster address: '{}' could not be resolved. Invalid or unresolveable address. Set 'syndic_master' value in minion config.�rMaster address: '{}' could not be resolved. Invalid or unresolveable address. Set 'master' value in minion config.�*��code�msgz'Master ip address changed from %s to %sZsource_interface_namez$Custom source interface required: %sz6The following interfaces are available on this Minion:ZupZinetZinet6�addressZ	source_ipzUsing %s as source IP addresszPThe interface %s is down so it cannot be used as source to connect to the Masterz&%s is not a valid interface. Ignoring.�source_addressZsource_ret_portz*Using %d as source port for the ret serverZsource_publish_portz*Using %d as source port for the master pubztcp://{ip}:{port})�ip�port�
master_urizMaster URI: %s)�get�salt.utils.networkr�utils�networkZ	dns_check�intrr�log�errorr&r�warning�format�trace�
interfaces�debug)�opts�fallback�retZ	check_dns�saltr.ZabortedZunknown_strr+�errrG�addrsrrr�resolve_dns�s���������
����
��


����
�

�
�rOc
Cszi}|ddkrt�|d�|d<|Sz
t|d�\}}Wnty-}zt|��d}~wwd|i}|r;|�d|i�|S)a
    parse host:port values from opts['master'] and return valid:
        master: ip address or hostname as a string
        master_port: (optional) master returner port as integer

    e.g.:
      - master: 'localhost:1234' -> {'master': 'localhost', 'master_port': 1234}
      - master: '127.0.0.1:1234' -> {'master': '127.0.0.1', 'master_port' :1234}
      - master: '[::1]:1234' -> {'master': '::1', 'master_port': 1234}
      - master: 'fe80::a00:27ff:fedc:ba98' -> {'master': 'fe80::a00:27ff:fedc:ba98'}
    Zmaster_uri_formatZip_onlyr+Nr-)r�
ip_addressr�
ValueErrorr�update)rIrK�hostr;�excrrr�prep_ip_port&s
���rUcKs�tj�|d�}|�dd�}|duri}nd|i}tj�|�s'tj|fi|��t�|�}|rGt�|j�}||dkrGt�	||j|A|dB�t
td�rr|�dd�}|�dd�}|j|ksb|j|krrdd	�||fD�rrt�
|||�|S)
a�
    Given the cache directory, return the directory that process data is
    stored in, creating it if it doesn't exist.
    The following optional Keyword Arguments are handled:

    mode: which is anything os.makedir would accept as mode.

    uid: the uid to set, if not set, or it is None or -1 no changes are
         made. Same applies if the directory is already owned by this
         uid. Must be int. Works only on unix/unix like systems.

    gid: the gid to set, if not set, or it is None or -1 no changes are
         made. Same applies if the directory is already owned by this
         gid. Must be int. Works only on unix/unix like systems.
    �proc�modeN�chown�uid����gidcSsg|]}|dkr|�qS)rZr)�.0�irrr�
<listcomp>rsz get_proc_dir.<locals>.<listcomp>)�os�path�join�pop�isdir�makedirs�stat�S_IMODE�st_mode�chmod�hasattr�st_uid�st_gidrX)�cachedir�kwargs�fn_rWZd_statZ	mode_partrYr[rrr�get_proc_dirDs(

�rocCsNtjj�|�}g}i}g}|D]p}t|t�rD|�dd�durD|��D] \}	}
|	dkr+q"|js3|	|jvr8|
||	<q"|�	|	�d|
���q"qtjjj
|gdd�d}|rz|js`tt|�
���|jvrf|�|�q|��D]\}	}
|�	|	�d|
���qjq|�	|�q|r�|s�tjj�|�|jr�t|t�r�|��D]\}	}
|
|d|	��<q�||fS)z{
    Detect the args and kwargs that need to be passed to a function call, and
    check them against what was passed.
    Z	__kwarg__FT�=)Z	conditionr0Z__pub_)rLr?�argsZget_function_argspec�
isinstance�dictr=�items�keywords�appendZparse_input�next�iter�keysrR�invalid_kwargs)�funcrq�dataZignore_invalidZargspec�_args�_kwargsrz�arg�key�valZstring_kwargrrr�load_args_and_kwargs{s:
��r�cCs�d|vrm|d}|�d�\}}z%tj�|||�}|st�||�|d<t|dtftf�s/t�d|d<Wn0tyKt	�
d|�t�tj
jj�Yntydt	�
d|d|�t�tj
jj�Ynwt	�d|�dSdS)	zd
    Evaluate master function if master type is 'func'
    and save it result in opts['master']
    Z__master_func_evaluatedr+�.TzFailed to load module %sz#%s returned from %s is not a stringz Evaluated master from module: %sN)�splitrL�loaderZraw_mod�KeyErrorrr�str�list�	TypeErrorrBrC�sys�exit�defaults�	exitcodes�
EX_GENERIC�info)rIZmod_fun�mod�funZ
master_modrrr�eval_master_func�s(��r�cCs>ddddd�}|dkr|dur|�|��d|��S|�|d�S)	z[
    Centralized master event function which will return event type based on event_map
    Z__master_connectedZ__master_disconnectedZ__master_failbackZ__master_alive)�	connected�disconnected�failback�aliver�N�_�r=)�typer+Z	event_maprrr�master_event�s�r�cCsdtjvrdSdS)z:
    Return the proper service name based on platform
    ZbsdZsalt_minionzsalt-minion)r��platformrrrr�service_name�sr�c@sZeZdZdd�Zddd�Zedd��Zd	d
�Zej	j
jjdd
d��Z
dd�Zddd�ZdS)�
MinionBasecCs||_|�dd�|_dS)N�beacons_leaderT)rIr=r���selfrIrrr�__init__�szMinionBase.__init__FNcCs4|duri}|r&tjj|j|jd|jd|jd|j�d�d���|jd<tjj|j|d�|_tjj|j|j|d	�|_	tj�
|j�|_
tjj|j|j	|d
�|_tjj|j|j	|jd�|_i|_
tjj|j|j	|j|j
|d�|_tjj|j|j	|d
�|_tj�|j�|_|j|j	d
<tjj|j|j	|j|d�|_dS)��
        Tell the minion to reload the execution modules

        CLI Example:

        .. code-block:: bash

            salt '*' sys.reload_modules
        N�grains�id�saltenv�	pillarenv�r��pillar)�context)r?r�)�	functionsr�)r��	returners)r�r?�serializersr��sys.reload_modules�r��proxyr�)rLr��
get_pillarrIr=�compile_pillarr�r?�minion_modsr�r�r�r��function_errors�states�render�rend�matchers�gen_modules�	executors)r��initial_loadr�rrrr��sN

��
�
���
��zMinionBase.gen_modulesc
Cs�z(t|d�r|j��nt�d�WdS|jj|kr&|jj}t�d�W|SW|StyA}z
t�d|�WYd}~|Sd}~ww)N�schedulezAMinion scheduler not initialized. Scheduled jobs will not be run.z3Overriding loop_interval because of scheduled jobs.z&Exception %s occurred in scheduled job)rir��evalrBrC�
loop_intervalrH�	Exception)�minionr�rTrrr�process_schedule s$
�����zMinionBase.process_schedulecCs<d|vr|dd|jddd�}|r|j�||jd�SgS)z|
        Evaluate all of the configured beacons, grab the config again in case
        the pillar or grains changed
        zconfig.merge�beaconsT)Z	omit_optsr�)rIr��process)r�r�Zb_confrrr�process_beacons3s�
�zMinionBase.process_beacons�<Tc#s���ddkrt�d�d|_tjjj�d��|���ddk�rj�ddk�rj�dd	kr5t	���n5�dd
v�rZt
�dt�r�t�d�d��dd
kr�t
�d�}|dkr��ddd�}t��d�|}z+�d��d|�d<�fdd��dD��dd�<��d<t�d�dd�Wntttfy�t�d�Ynwt�d��dr�t�d��d�d<d�d<�drшddkrшd�d<nrt
�dt�r�d�vr�dg�d<n_�ddkr�t�d�d�nP|�r2|�r�d�d<nCt�d �d��d�d!v�r+�d}	�fd"d��d!D��d<�d�|	�n�d�d<nt�d#t�d��t�tjjj��d$�rY�dd%k�rYt�d&�d�d$<nt�d'�d�t�tjjj�||d(�}
t|d)d��r{|j|
d)<��d*d�}d}t
�dt��r)d}
d}g�d+<t � �d��d!<�d�rʈdd%k�rÈd�rÈd!dd�}t!�"|�|�d!dd�<nt!�"�d!��d!D];}|�d<��#t$����dd%k�r�z�d+�t%�d�d,�W�q�t&�y�Y�q�w�d+�t%��d,��qΈd+�sd-}t�|�t&|��d}	|dk�r-tjjj�'�d/�V|d7}|dk�r>t�(d0||�nt�(d1|��d!D]�}|�d<��#t$����dd%k�rrz
��#t%�d��Wnt&�yqY�qHw��#t%���d�v�r�t � �d!��d<�|_)tj*j+j,j-�fi|
��}zNz|�.�Vd.}
|}d}WW|�r�|�/�n@t&�y�}z*|}|j0�1d2��r�t�d3�d|�nt�d4�d�WYd}~W|�r�|�/��qHd}~ww|�r�|�/�ww|
�r�|}|
�s||k�rd|_t � |j)d!�|j)d<t�d5�|�n|j2�3d6�|_4d.|_tjjj��d|f���q�d�r3t�d7�d}	|dk�rFtjjj�'�d/�V|d7}|dk�rWt�(d0||�nt�(d1|���#t$�����#t%���zi|j)d8d9k�r�d.|j)d:<tj5j6D].}|d;k�r�t7�s��q}||j)d8<tj*j+j,j-|j)fi|
��}|�.�V|j2j8�s��q}|j)d:=ntj*j+j,j-|j)fi|
��}|�.�V|j2�3d6�|_4d.|_tjjj��d|f��t&�y�|�r�|�/�||k�r�d|_�Ynw�q6)<a
        Evaluates and returns a tuple of the current master address and the pub_channel.

        In standard mode, just creates a pub_channel with the given master address.

        With master_type=func evaluates the current master address from the given
        module and then creates a pub_channel.

        With master_type=failover takes the list of masters and loops through them.
        The first one that allows the minion to create a pub_channel is then
        returned. If this function is called outside the minions initialization
        phase (for example from the minions main event-loop when a master connection
        loss was detected), 'failed' should be set to True. The current
        (possibly failed) master will then be removed from the list of masters.
        �master_type�disablez-Master is set to disable, skipping connectionF�NNr�r1r2r{��failover�distributedr+z*Got list of available master addresses: %sr�r0Nr�rcsg|]
}|�dkr|�qS)rr)r\�m)�preferred_mastersrrr^rs
�z*MinionBase.eval_master.<locals>.<listcomp>z"Distributed to the master at '%s'.z*Failed to distribute to a specific master.z3master_type = distributed needs more than 1 master.Zmaster_shufflez�Use of 'master_shuffle' detected. 'master_shuffle' is deprecated in favor of 'random_master'. Please update your minion config file.Z
random_masterZ
auth_tries�master_failback�master_failback_interval�master_alive_interval�master_listz$Syndic setting master_syndic to '%s'zBMoving possibly failed master %s to the end of the list of mastersZ
local_masterscsg|]
}�d|kr|�qS)r+r�r\�x�rIrrr^�szMmaster_type set to 'failover' but 'master' is not of type list but of type %sr/r�z�'master_type' set to 'failover' but 'retry_dns' is not 0. Setting 'retry_dns' to 0 to failover to the next master on DNS errors.z/Invalid keyword '%s' for variable 'master_type')�timeout�safe�io_loopZmaster_triesZmaster_uri_listr<zNo master could be resolvedT�acceptance_wait_timez&Connecting to master. Attempt %s of %sz4Connecting to master. Attempt %s (infinite attempts)zCould not accessz\Failed to initiate connection with Master %s: check ownership/permissions. Error message: %sz;Master %s could not be reached, trying next master (if any)zQNo master could be reached or all masters denied the minion's connection attempt.ssaltzGrandom_master is True but there is only one master specified. Ignoring.�	transportZdetectZdetect_modeZzeromq)9rBrDr�rL�ext�tornado�gen�Return�_discover_mastersr�rrr�r��len�binascii�crc32r��AttributeErrorr�r�rvrCr�r�r�r�r�r��critical�getattrr�r=�copy�random�shufflerRrUrOrr"rHrI�channel�clientZAsyncPubChannel�factory�connect�close�strerror�
startswith�authZ	gen_token�tokr�Z
TRANSPORTSrZ
authenticated)r�rIr�r��failedr�Z
master_lenZsecondary_mastersZ
master_idxZ
failed_masterZfactory_kwargs�tries�attempts�conn�last_excr+r7�pub_channelZret_pub_channelrTZtransr)rIr�r�eval_masterBs��
�
�
��������

�
����




��



��

��
������
�
���
T�

�


��

�
��

���zMinionBase.eval_masterc
s�|jdtdkr�|jddur�tjj��}i}t|jd�dd��D]@}z%|d7}t�	d|�|�
|���|sFt�
|jd�dd	��nWnWq#tyc}z
t�d
|�WYd}~nd}~ww|r�|j�di��dd
�}|dvr}t�d|�dS|jd�di�}|��D]5\}}|D](�t�fdd�|��D��}	|d
kr�t|	�s�|	t|�kr��d|jd<dSq�q�dSdSdSdS)z�
        Discover master(s) and decide where to connect, if SSDP is around.
        This modifies the configuration on the fly.
        :return:
        r+Z	discoveryFr��r0z)Attempting %s time(s) to discover masters�pause�zSSDP discovery failure: %sN�match�any)r��allzPSSDP configuration matcher failure: unknown value "%s". Should be "any" or "all"�mappingcs*g|]\}}��di��|�|kr|�qS)r�r�)r\r��value�Z
proto_datarrr^�s
�z0MinionBase._discover_masters.<locals>.<listcomp>)rIrrLr?ZssdpZSSDPDiscoveryClient�ranger=rBr�rRZdiscoverr r"r�rCrtr��bool)
r�Zmaster_discovery_client�mastersZattrM�policyr��addrZmappingsZcntrr�rr��sT����
�����
zMinionBase._discover_mastersc	Cs�d}|j�d�rM|r|jdSzt�|jd|jd�}||}t�d||�|WStyLt�d|jd|jd�t�|td�tdYSwt�||j�d��|j�d�S)z�
        Based on the minion configuration, either return a randomized timer or
        just return the value of the return_retry_timer.
        z+Minion return retry timer set to %s secondsZreturn_retry_timer_maxZreturn_retry_timerz%s (randomized)zeInvalid value (return_retry_timer: %s or return_retry_timer_max: %s). Both must be positive integers.)	rIr=r��randintrBrHrQrCr)r��maxr7Zrandom_retryZ	retry_msgrrr�_return_retry_timer�s,
���zMinionBase._return_retry_timer�FN)r�TFF�F)�__name__�
__module__�__qualname__r�r��staticmethodr�r�rLr�r�r��	coroutiner�r�rrrrrr��s
3

C0r�cs"eZdZdZd�fdd�	Z�ZS)�SMinionz�
    Create an object that has loaded all of the minion module functions,
    grains, modules, returners etc.  The SMinion allows developers to
    generate all of the salt minion functions and present them with these
    functions for general use.
    Ncs�ddl}|j�|�|d<t��|��j�dd�dks"�j�dd�r?|jjj	j
��}|jjjj
�fdd���|��fd	d
���jd|d��jddkr�j�d
d�r�ddl}tj��jdd�}tj�|�snt�|d�tj�|d�}�jddur��jd}nd}|�jddgii}|jj�|d��}	|jjj||	td�t�|d�Wd�n1s�wYtj�|d�}
|jj�|
d��}	|jjj�jd|	td�t�|
d�Wd�dS1s�wYdSdSdS)Nrr�r'r(r*Fc3s$��j�jdd�V\}}|��dS)zb
                Wrap eval master in order to close the returned publish channel.
                T�r�N)r�rIr�)r+r��r�rrr��s�z%SMinion.__init__.<locals>.eval_mastercs��S�Nrr)r�rr�<lambda>�sz"SMinion.__init__.<locals>.<lambda>T)r�r�Zminion_pillar_cacherlr�i�ztop.slsr��baser��cache�wb)�encodingi�z	cache.sls)�salt.loaderr�r��superr�rIr=r�r��ioloop�IOLoop�currentr�rZrun_syncr�Zsalt.utils.yamlr_r`rarcrdr?�files�fopenZyamlZ	safe_dumprrh)r�rIr�rLr�ZpdirZptopZpenvZ	cache_top�fp_Z	cache_sls��	__class__)r�r�rr��sJ�

����"��zSMinion.__init__r)r
rr�__doc__r��
__classcell__rrr rr�src@sHeZdZdZ						ddd�Zdd�Zdd	�Zd
d�Zdd
d�ZdS)�MasterMinionz�
    Create a fully loaded minion function object for generic use on the
    master. What makes this class different is that the pillar is
    omitted, otherwise everything else is loaded cleanly.
    TNcCsjd|_d|_tjj|d||d�|_||_||_||_||_	||_
d|_d|_d|_
d|_|jdd�dS)NZ	conf_file)�ignore_config_errorsT)r�)r�r�rLZconfigZmminion_configrI�	whitelist�mk_returners�	mk_states�mk_rend�
mk_matcherr�r�r?r�r�)r�rIr�r�r��matcherr&r%rrrr�s

�zMasterMinion.__init__c	Cs|jdur2|jD]}z|j|}t|d�r|��Wqty#Yqwt|jd�r/|j��i|_|jdurBt|jd�rB|j��i|_|jdurUt|jd�rU|j��i|_t|d�rp|jdurpt|jd�rm|j��i|_t|d�r�|jdur�t|jd�r�|j��i|_t|d�r�|jdur�t|jd�r�|j��i|_t|d�r�|j	dur�t|j	d�r�|j	��i|_	t|d�r�|j
dur�t|j
d�r�|j
��i|_
t|d�r�|jdur�t|jd�r�|j��i|_|j�r	d	|jv�rt|jd	d��r|jd	��i|jd	<dSdSdS)
z1
        Destroy the MasterMinion object
        N�destroyr�r�r�r�r�r�r�)
r�rir,r�r�r?r�r�r�r�r�r�rI)r��returnerr{rrrr,2s`



��








�zMasterMinion.destroycCs|Srrrrrr�	__enter__eszMasterMinion.__enter__cG�|��dSr�r,)r�rqrrr�__exit__h�zMasterMinion.__exit__FcCs�tj�|j�|_tjj|j|j|j|d�|_tj�|j�|_|jr+tj�	|j|j�|_	|j
r<tj�|j|j|j|j�|_|jrItj�
|j|j�|_|jrTtj�|j�|_|j|jd<dS)r�)r?r&r�r�N)rLr�r?rIr�r&r�r�r'r�r(r�r)r�r�r*r�r�)r�r�rrrr�ks&
��zMasterMinion.gen_modules)TTTTNTr	)	r
rrr"r�r,r.r1r�rrrrr$s	
�3r$cs�eZdZdZ�fdd�Zdd�Zdd�Zejj	j
jdd	��Zd
d�Z
				
ddd�Zdd�Zd dd�Zejj	j
jdd��Zdd�Zedd��Zdd�Zejj	j
jdd��Z�ZS)!�
MinionManagerz�
    Create a multi minion interface, this creates as many minions as are
    defined in the master option and binds each minion object to a respective
    master.
    cszt��|�|jd|_|jd|_g|_g|_tjj	j
j��|_
tdd�|_|j
j|jjfiddi��d|_d|_dS)Nr��acceptance_wait_time_maxZMultiMinionProcessManager��name�asynchronousT)rr�rI�	auth_wait�
max_auth_wait�minions�	jid_queuerLr�r�rrrr�r�process_manager�spawn_callback�run�event_publisher�eventr�r rrr��s��
zMinionManager.__init__cCr/rr0rrrr�__del__�r2zMinionManager.__del__cCsPtjjj|j|jd�|_tjjjd|j|jd�|_|j�d�|j�	|j
�dS)N�r�r�)rIr�r,)rLr?r@ZAsyncEventPublisherrIr�r?�	get_event�	subscribe�set_event_handler�handle_eventrrrr�_bind�s�
�zMinionManager._bindc
#sR�z�fdd�|jD�VWdSty(}z
t�d|�WYd}~dSd}~ww)Ncsg|]}|����qSr)rF)r\r���packagerrr^�sz.MinionManager.handle_event.<locals>.<listcomp>zError dispatching event. %s)r:r�rBrC)r�rIrTrrHrrF�s���zMinionManager.handle_eventcCs^t|d�r|jdur|j��|j��t|d�r-|jD]}t|d�r'|��qg|_dSdS)z-
        Tear down the MinionManager
        r<Nr:r,)rir<�stop_restarting�
kill_childrenr:r,�r�r�rrrr,�s




�
�zMinionManager.destroyNTc	Cst|||||||d�S)�F
        Helper function to return the correct type of object
        �r��loaded_base_namer;�load_grains)�Minion�r�rIr�r�r�rOr;rPrrr�_create_minion_object��
�z#MinionManager._create_minion_objectcCs|js
t�d�dSdS)zQ
        Check the size of self.minions and raise an error if it's empty
        z7Minion unable to successfully connect to a Salt Master.N)r:rBrCrrrr�_check_minions�s�zMinionManager._check_minionsr�c
Cs�|jd}|jddvst|jdt�s|g}d}|D]4}t�|j�}||d<d|d<||d<|r3d}|j||dd|jd	�|d�|jd
�}|j�	|j
|�q|j�||j�dS)zH
        Spawn all the coroutines which will sign in to masters
        r+r�r�T�multimasterr�FZauth_timeoutzsalt.loader.{})r�rOr;N)
rIrrr�r��deepcopyrSr�rEr;r=�_connect_minion�
call_laterrU)r�r�rr�r+�s_optsr�rrr�_spawn_minions�s.

��zMinionManager._spawn_minionsc
cs��d}|jd}d}	z9|j�dd�r|jdd�|j�dd�r&|jdd�|j�dd	�d
kr6|j|d�V|jdd�|j�|�WdStyz}z*|�	�Vd}t
�d
|jd|�||jkrg||j
7}tjjj�|�VWYd}~nVd}~wty�|�	�Vd�|jd�}t
�|�YdSty�}z+|�	�Vd}t
jd|jddd�||jkr�||j
7}tjjj�|�VWYd}~nd}~wwq)zL
        Create a minion, and asynchronously connect it to a master
        rr�FT�beacons_before_connect��before_connect�scheduler_before_connectr�r�r�r)�startzeError while bringing up minion for multi-master. Is master at %s responding? The error message was %sr+Nr3�'Unexpected error while connecting to %s��exc_info)rIr=�
setup_beacons�setup_scheduler�connect_master�tune_inr:rvr�
destroy_asyncrBrCr9r8rLr�r�r�r"rrEr�r�)r�r��lastr8r�rTrM�errrrXsd�

�

�
��

�

���zMinionManager._connect_minioncCs|��|��|j��dS)a
        Bind to the masters

        This loop will attempt to create connections to masters it hasn't connected
        to yet, but once the initial connection is made it is up to ZMQ to do the
        reconnect (don't know of an API to get the state here in salt)
        N)rGr[r�r`rrrrrg:szMinionManager.tune_incCs|jD]}|jrdSqdS)NTF)r:�restartrLrrrrkJs

�zMinionManager.restartcCst�|j�|j||�dS)z�
        Stop minions managed by the MinionManager

        Called from cli.daemons.Minion._handle_signals().
        Adds stop_async as callback to the io_loop to prevent blocking.
        N)rr��add_callback�
stop_async)r��signum�parent_sig_handlerrrr�stopQs
�zMinionManager.stopccs��tjjj�d�V|jD]}|j��|j�|�|j�	�|�
�Vq
|jdur2|j��d|_|j
dur?|j
��d|_
||d�dS)z�
        Stop minions managed by the MinionManager allowing the io_loop to run
        and any remaining events to be processed before stopping the minions.
        r�N)rLr�r�r�r"r:r<rJZsend_signal_to_processesrKrhr?r�r@r,)r�rnror�rrrrmcs�






zMinionManager.stop_async�NNNT)r�)r
rrr"r�rArGrLr�r�r�rrFr,rSrUr[rXrg�propertyrkrprmr#rrr rr3�s.


�


8

r3cs$eZdZdZ						d��fdd�	Zdd�Zd	d
�Zd�dd
�Zej	j
jjd�dd��Z
ej	j
jjdd��Zej	j
jjdd��Zdd�Z	d�dd�Zdd�Zej	j
jjdd��Zej	j
jjdd��Z	d�dd�Zej	j
jj							d�d d!��Z							d�d"d#�Zej	j
jjd$d%��Zd&d'�Zd(d)�Zd�d*d+�Zd,d-�Zd.d/�Zd0d1�Zd2d3�Zd4d5�Z ej	j
jjd6d7��Z!d8d9�Z"d:d;�Z#e$d<d=��Z%d>d?�Z&e$d@dA��Z'e$dBdC��Z(d�dEdF�Z)ej	j
jjd�dGdH��Z*d�dIdJ�Z+d�dKdL�Z,dMdN�Z-dOdP�Z.ej	j
jjdQdR��Z/d�dSdT�Z0dUdV�Z1dWdX�Z2dYdZ�Z3ej	j
jjd�d[d\��Z4d]d^�Z5d_d`�Z6dadb�Z7dcdd�Z8dedf�Z9ej	j
jjdgdh��Z:didj�Z;dkdl�Z<d�dmdn�Z=d�dodp�Z>dqdr�Z?dsdt�Z@ej	j
jjdudv��ZAdwdx�ZBd�dzd{�ZCd|d}�ZDd�d~d�ZEd�d��ZFd�d��ZGej	j
jjd�d���ZHd�d��ZId�d��ZJ�ZKS)�rQz}
    This class instantiates a minion, runs connections for a minion,
    and loads all of the functions into the minion
    r�TNc
s:t��|�tjj�|j�}tj�	|�r+z
t�
|�t�d|�Wn	t
y*Ynw|��||_||_d|_tjj��|_||_d|_d|_d|_|durQgn||_i|_d|_d|_|durktjjjj �!�|_"n||_"t#ryt$dkryt�%d�tjj&�'�s�|r�tj(�)|�|jd<n"|j�*dd�r�t�%d	�d|jd<|j�*d
d�r�t�%d�d|jd
<t�d�|jd
r�t+�,d|jd
�}	t�d|	|jd
�t-�.|	�t/dd�|_0|j"j1|j0j2fiddi��tjj&�'�s�|j"�1tj3j4|j|j0�t5�6t5j7�t5j8u�rt5�5t5j7|j9�t5�6t5j:�t5j8u�rt5�5t5j:|j9�dSdS)z*
        Pass in the options dict
        zRemoved stale lock file: %sNFr)r��z�You have a version of ZMQ less than ZMQ 3.2! There are known connection keep-alive issues with ZMQ < 3.2 which may result in loss of contact with minions. Please upgrade your ZMQ!r�r\zM'beacons_before_connect' is not supported for proxy minions. Setting to Falser_zO'scheduler_before_connect' is not supported for proxy minions. Setting to FalsezCreating minion process managerZrandom_startup_delayzWMinion sleeping for %s seconds due to configured startup_delay between 0 and %s secondsZMinionProcessManagerr5r7T);rr�rLr?�stateZqueue_lock_pathrIr_r`�isfile�removerBr��OSError�_cleanup_orphaned_queue_filesr�r��_runningr�ZSubprocessList�subprocess_listrOr�rk�readyr;�periodic_callbacks�req_channel�$_system_resource_limit_hit_timestampr�r�rrrr�rrrDr�Zis_proxyr�r�r=r�rr r"rr<r=r>ZenginesZ
start_engines�signal�	getsignal�SIGINT�SIG_DFL�_handle_signals�SIGTERM)
r�rIr�r�rOr�r;rPZ	lock_pathZ
sleep_timer rrr��s�
��
��
�


�
��zMinion.__init__cCs,d|_|j�||�t�d�t�d�dS)NFr0r)ryr<r�r r"r�r�)r�rnZsigframerrrr��s
zMinion._handle_signalscCstjj�|j�tjj�|j�fD]u}tj�|�sqzSt�	|�D]K}|�
d�r+|�d�s,qtj�||�}zt�
|�}t��|j}|dkrPt�d||�t�|�Wqtyj}z
t�d||�WYd}~qd}~wwWqty�}z
t�d||�WYd}~qd}~wwdS)z�
        Clean up orphaned running_ queue files that may have been left behind
        if the minion crashed after renaming queued_ to running_ but before cleanup.
        �running_�.pi,z5Removing orphaned running queue file: %s (age: %.1fs)z+Could not check/cleanup running file %s: %sNz%Could not list queue directory %s: %s)rLr?rt�state_queue_dirrI�
job_queue_dirr_r`�exists�listdirr��endswithrarer �st_mtimerBr�rvrwrH)r��	queue_dir�fnr`�	stat_infoZage_secondsrTrrrrxsB�
�
��������z$Minion._cleanup_orphaned_queue_filesFcs�d�_t�d��fdd�}�j|d��_�j�|�|r&�j�|�jj�z�j�	�Wn
t
y:���Ynw�j��rX�j�
�}|rX�j��}|d�|d|d��|rc�jduretd	��d
Sd
S)z:
        Block until we are connected to a master
        F�sync_connect_mastercsd�_�j��dS)NT)�_sync_connect_master_successr�rp)�futurerrr�on_connect_master_future_done/szAMinion.sync_connect_master.<locals>.on_connect_master_future_donerrr0rsz$Failed to connect to the salt-masterN)r�rBrHrfZ_connect_master_futureZadd_done_callbackr�rYrpr`�KeyboardInterruptr,�done�	exceptionrc�with_tracebackr	)r�r�r�r�Zfuture_exceptionrcrrrr�(s(
�


�zMinion.sync_connect_masterccs��t|d�r&|jr&|j�d�t|jd�r|jj��t|jd�r&|j��t|d�r@|jr@|j��Vtj	j
j�d�Vd|_|�
|j|j|j|�V\}|_tjjjj|j|jd�|_t�d�|j��V|�|�VdS)	zX
        Return a future which will complete when you are connected to a master
        r�Nr�r�r}rrBz,Connecting minion's long-running req channel)rir��on_recvr��
invalidater�r}�close_asyncrLr�r�r�r"r�rIr�r�r�r��AsyncReqChannelr�r�rBrHr��_post_master_init)r�r�r+rrrrfHs(�
�
�
zMinion.connect_masterccs&�|j�|�||�V|j��dSr)ZpayloadsrvZpayload_ack�notify)r��payloadZ
reply_funcrrr�handle_payloades�
zMinion.handle_payloadccsr�|jr.||jd<tjj|j|jd|jd|jd|j�d�d�}|��V|jd<|��|js6|�	�n#|jrY|jdrY|�
�\|_|_|_
|_t|d�rY|j|j_|j|j_t|d�sqtjjj|j|j|jtd	d
�gd�|_|jdr�d
|jvr�|jjdd
|jdddd|j�dd�d�idd�t�d�n|jjddd�|jddk�r|jddk�r|j�r|jjtd	|jdd�d|jdddd|jddd�d�idd�|jd �rd!|jv�r|jd|jd!dk�r|jjtd"d
�d#|jd$dddd|jd!did�idd�d%S|jjtd"d
�dd�d%S|jjtd	|jdd�dd�|jjtd"d
�dd�d%S)&a�
        Function to finish init after connecting to a master

        This is primarily loading modules, pillars, etc. (since they need
        to know which master they connected to)

        If this function is changed, please check ProxyMinion._post_master_init
        to see if those changes need to be propagated.

        Minions and ProxyMinions need significantly different post master setups,
        which is why the differences are not factored out into separate helper
        functions.
        r+r�r�r�r�r�r�r�r��r�)�cleanupZmine_enabledzmine.updateZ__mine_intervalZ
mine_intervalTrsZmine_return_jobF)�function�minutes�jid_include�
maxrunningZrun_on_start�
return_job)�persistzAdded mine.update to schedulerr��tcpr�r�r�r+�
status.masterr0�r+r��r��secondsr�r�r�rmr�r�r��status.ping_masterr�N)r�rIrLr��get_async_pillarr=r�r,r{�_setup_core�
_load_modulesr�r�r�r�rir�r?�Scheduler��add_jobrBr��
delete_job)r�r+�async_pillarrrrr�ls��

�
�




���
�
���
����

��zMinion._post_master_initcCs.i}|j��D]
\}}|dkrq|||<q|S)zG
        Returns a copy of the opts with key bits stripped out
        �logger)rIrt)r��mod_optsr�r�rrr�_prep_mod_opts�s
zMinion._prep_mod_optscCs�d}|s	|j}d}d}d}|�dd�dkrJtrJtrJt�d|d�d}t�tj�}t	�
t����
�dd�\}	}
|	|
|d}t�tj||f�n|�dd�dkr`tsYt�d	�ts`t�d
�t|d�ri|j}nd}|durqi}|dur�tjj||||d�|d
<tjj|||d�|_|�dd�r�t�|�}
tjj|
|j||j||d�}ntjj||j|||d�}tjj||||d�}i}d|vr�|d}|�d�|dur�t�tj|�tjj||||d�}|r�||_||||fS)za
        Return the functions and the returners loaded up from the loader
        module
        TFN�modules_max_memoryrZrz1modules_max_memory set, enforcing a maximum of %srsz>Unable to enforce modules_max_memory because psutil is missingz@Unable to enforce modules_max_memory because resource is missingr��r�r�r�rV)r?r�rOr�r��r?r�r�r��_errors)rIr=�
HAS_PSUTIL�HAS_RESOURCErBrH�resource�	getrlimitZ	RLIMIT_AS�psutil�Processr_�getpidZmemory_infoZ	setrlimitrCrir�rLr�r�r?r�rWr�rOr�rbr�)r��
force_refreshr�r�rIr�Zopt_inr�Z
old_mem_limitZrssZvmsZ	mem_limitr�rZr�r��errorsr�rrrr��s~���

�
�	�
zMinion._load_modulescCs�tjjjd|jdd��Q}tt���}t�	d|�|j
|d|�d|jd��|d�|jd	|��|d
�}|rV|�d�rEtj�
d|d����t�	d
|�|dWd�Stj�
d��1s_wYdS)Nr�T�rI�listen�Send request to main id=%s�__master_req_channel_payload/�/r+�r��__master_req_channel_return/)�tag�waitrCz#Request timed out in main process: �Reply from main %srKzRequest timed out)rLr?r@rCrIr��uuid�uuid4rBrF�
fire_eventr=�
exceptionsr
)r��loadr�r@�
request_idrKrrr�_send_req_syncAs,��
���zMinion._send_req_syncccs��tjjjd|jdd��]}tt���}t�	d|�|j
|d|�d|jd��|d�Vt��}t��||krU|jd	|��dd
�}|rDntjj
j�d�Vt��||ks7td��t�	d
|�tjj
j�|d��1slwYdS)Nr�Tr�r�r�r�r+r�r�)r�Zno_block�333333�?zDid not recieve return eventr�rK)rLr?r@rCrIr�r�r�rBrF�fire_event_asyncr r�r�r�r"�TimeoutErrorr�)r�r�r�r@r�r`rKrrr�_send_req_asyncWs,��
���zMinion._send_req_asyncccs,�|jj|||jdd�V}tjjj�|��)z�
        Send a request to the master's request server. To be called from the
        top level process in the main thread only. Worker threads and
        processess should call _send_req_sync or _send_req_async as nessecery.
        �return_retry_tries�r�r�)r}�sendrIrLr�r�r�r��r�r�r�rKrrr�_send_req_async_mainos
��zMinion._send_req_async_maincs��jdd|�jd�}|r||d<n|r|r||d<||d<n|s,|r,i|d<||d<ndS|rD�fdd��j�d	i���D�}||d	<|S)
Nr�Z
_minion_event)r��cmd�pretagr��eventsr|r�cs$i|]\}}|�jdvr||�qS)�start_event_grainsr�)r\�k�vrrr�
<dictcomp>�s
�z/Minion._fire_master_prepare.<locals>.<dictcomp>r�)rIr�r=rt)r�r|r�r�r��include_startup_grainsr�Z
grains_to_addrrr�_fire_master_prepare{s(�



�zMinion._fire_master_preparec
cs:�|�|||||�}|durdd�}	|	}|�||�VdS)NcWst�d�dS)N�Efire_master failed: master could not be contacted. Request timed out.T)rBr��r�rrr�handle_timeout�s�z0Minion._fire_master_main.<locals>.handle_timeout)r�r�)
r�r|r�r�r�r��timeout_handlerr�r�r�rrr�_fire_master_main�s�
�zMinion._fire_master_mainc	Csj|�|||||�}z	|�||�WdStjjy#t�d�YdSty4t�dt�	��YdSw)zQ
        Fire an event on the master, or drop message if unable to send.
        r�Fzfire_master failed: %sT)
r�r�rLr�r
rBr�r��	traceback�
format_exc)	r�r|r�r�r�r�r�r�r�rrr�_fire_master�s
�
���zMinion._fire_masterccs�|�|�VdS)�b
        Override this method if you wish to handle the decoded data
        differently.
        N)�_handle_decoded_payload_impl�r�r|rrr�_handle_decoded_payload�s�zMinion._handle_decoded_payloadc	�s��d|vrt�d|d|d|d�nt�d|d|d�t�d|�t�d|j�|�dd	�}|jd
ur[|d|jvrC|sBd
Sn|j�|d�t|j�|jdkr[|j�	d�t
|dt�r~|dd
kr~|��\|_
|_|_|_|j
|j_
|j|j_|j�d�r�t|d�r�|j}nd
}tjj|jd|d�|jd<|�	dd	�}|s�|��s�t�d|d�|�|�d
S|��s�t�d|d�|�|�d
S|�sUdd�|jjD�}t|�}|��}||k�rUzWtjj� |j�4Id
H�=dd�|jjD�}t|�}|��}||k�r#t�d|d||�|�|�	Wd
�Id
HWd
SWd
�Id
Hn1Id
H�s4wYWntj!j"�yTt�d|d�|�|�Yd
Sw|�#|�}|�r�t$j%�&|jdd�}	t$j%�'|	��s�zt$�(|	�Wn
t)�yYnwt$j%�&|	t|d��}
t*|dt$�+��}|d
u�r�t$�+�}|�,�}||d<z"tjj-�.|
d��}
tj/�0||
�Wd
�n	1�s�wYWnt)�y�t�1d|
�YnwdD]}||v�r�z
t$�2||�W�q�t)�y�Y�q�w�q�d
S) zA
        Async implementation of _handle_decoded_payload
        �userz(User %s Executing command %s with jid %sr��jidz Executing command %s with jid %szCommand details %szStarted JIDs: %s�__ignore_process_count_maxFNZminion_jid_queue_hwmrr�Zgrains_refresh_pre_execr�T)r�r�r�z(Insufficient FD headroom, queuing job %sz,Insufficient memory headroom, queuing job %scS�g|]}|��r|�qSr��is_alive�r\�prrrr^�
��z7Minion._handle_decoded_payload_impl.<locals>.<listcomp>cSr�rr�r�rrrr^)r�z]Maximum number of processes reached while executing jid %s, queuing... (Running: %s, Max: %s)z<Failed to acquire job_queue lock for jid %s, queuing anyway.rlrV�pid�w+bz(Failed to write placeholder proc file %s)�_job_queue_file�_state_queue_file)3rBr�rHrFr;r=rvr�rIrbrrr�r�r�r�r�r�r�rir�rLr�r��_has_fd_headroomrD�
_queue_job�_has_memory_headroomrz�	processes� _get_effective_process_count_maxr?rt�acquire_async_queue_lockr��
FileLockError�_invoke_executionr_r`rarcrdrwr�r�r�rrr��dumprCrv)r�r|Zbypass_checkr�Zbypass_process_count_max�running_processes�
process_count�process_count_maxrV�proc_dirZproc_fnZreal_pidZplaceholder_datarZqfrrrr��s���
��


�

�
�
�
�*���
�
�
���
���z#Minion._handle_decoded_payload_implcCstjj�|j�}tj�|�szt�|�Wn	t	yYnw|�
d�}dtt��d��d|�d�}tj�
||�}z5|d}tjj�|d��}tj�||�Wd�n1sYwYtjj�||�t�d	||�WdSt	yt�d
|�YdSw)zK
        Queue a job to disk because process_count_max is reached.
        r��queued_i@Br�r�z.tmpr�NzQueued job %s to %sz!Failed to write job queue file %s)rLr?rtr�rIr_r`r�rdrwr=rAr rarrr�r
Z
atomicfileZ
atomic_renamerBr�rC)r�r|r�r�r�r`Ztmp_pathrrrrris(�
��zMinion._queue_jobcCs�tsdSzit�tj�\}}|tjkrWdStj�d�r#tt�	d��}nzddl
}|��}|��}Wnt
y<YWdSw||dkrMt�d||�WdS|sk|j�dd�skt|d	�}||krkt�d
|||�WdSWdSt
ywYdSw)z�
        Check if we have enough file descriptors available to start a new process safely.
        Returns True if we have headroom, False otherwise.
        Tz
/proc/self/fdrN�dzJFD critical limit reached (Open: %s, Limit: %s). Pausing queue processing.F�multiprocessingg�������?zPFD limit reached (Open: %s, Limit: %s, Threshold: %s). Pausing queue processing.)r�r�r��
RLIMIT_NOFILE�
RLIM_INFINITYr_r`rcr�r�r�r�Znum_fdsr�rBrDrIr=rA)r�Z
critical_only�
soft_limitr�Zcurrent_fdsr�r��limitrrrr�sJ
����zMinion._has_fd_headroomcCsRtsdSzddl}|��}|jdkrt�d|j�WdSWdSty(YdSw)z�
        Check if we have enough memory to start a new process.
        Returns True if we have headroom, False otherwise.
        TrN�_z<Memory limit reached (Used: %s%%). Pausing queue processing.F)r�r�Zvirtual_memory�percentrBrDr�)r�r�Zmemrrrr�s 
����zMinion._has_memory_headroomcCs�|j�d�}|dur|dkr|StsdSzt�tj�\}}|tjkr%WdSWn
ty0YdSw|j�dd�}|r=dnd	}d}|||}td
|�S)z�
        Calculate the effective process_count_max.
        If configured value is > 0, use it.
        If <= 0 (unlimited), calculate a safe limit based on RLIMIT_NOFILE.
        r
Nrr��rT�r�r0)	rIr=r�r�r�rrr�r)r�rrr�Zis_threadingZfds_per_jobZreservedZ
safe_limitrrrr�s$
��
z'Minion._get_effective_process_count_maxcCs�|}d}|j�dd�}d�|d�}|rHtjj��r!d}tjjj	}t
tjtj
��t|j|||j||j|fd�}Wd�n1sBwYntj|j||j||j|f|d�}|r�t
tjtj
��Iz|��Wn:ty�}z.|jtjtjtjtjfvr�t�d|d|�t��|_|�|�WYd}~Wd�dS�d}~wwWd�n1s�wYn;z|��Wn4ty�}z(|jtjtjtjtjfvr�t�d|d|�t��|_|�|�WYd}~dS�d}~ww|j� |�|S)	NrTzProcessPayload(jid={})r�)�targetr6rq)rrqr6zWSystem resource limit reached while starting process for jid %s, queuing job. Error: %szVSystem resource limit reached while starting thread for jid %s, queuing job. Error: %s)!rIr=rErLr?r�Zspawning_platform�crypt�	AsyncAuth�	creds_maprrr�r�r�_targetr��	threading�Threadr`rw�errno�EAGAIN�EMFILE�ENFILE�ENOMEMrBrDr r~rrz�add)r�r|�instancerZmultiprocessing_enabledr6r�rTrrrr	�s�
������


��������

��zMinion._invoke_executioncC�$d|jvr|jd|jdd�dSdS)z6
        Set up the process queue processing.
        Z
process_queuer���intervalN)r|�add_periodic_callback�process_process_queuerrrr�setup_process_queue_processing8	s


��z%Minion.setup_process_queue_processingcC�(t|dd�rdSd|_|j�|j�dS)z�
        Check the process queue for pending jobs and execute them if slots are available.
        Runs as an async task on the main loop.
        � _process_queue_processing_activeFNT)r�r/r�r=�_process_process_queue_asyncrrrrr,A	�zMinion.process_process_queuecc��|��VdS)z6
        Async body of process_process_queue.
        N)�!_process_process_queue_async_implrrrrr0L	��z#Minion._process_process_queue_asyncc
�s���zC�z$tjj�|j�}tj�|�sWWd|_dS|�	�}�z�tjj�
|j�4IdH���dd�|jjD�}t
|�}|�	�}||krU	Wd�IdHWWWd|_dS|jdkr}t��|jdkrzt�d�	Wd�IdHWWWd|_dSd|_|��s�t�d�t��|_	Wd�IdHWWWd|_dSg}zt�|�D]}|�d	�r�|�d
�r�|�|�q�Wn	ty�Ynw|s�	Wd�IdHWWWd|_dSt�dt
|�||�|��|dkr�td||�}nd}|d|�D]�}tj�||�}t
d
d�|jjD��}	|�	�}
|	|
k�rt�d|	|
�n�z!tjj�|d��}tj�|�}Wd�n	1�s9wYWnJtt f�y�}
z;t!|
t��rg|
j"t"j#t"j$fv�rgt�%d|�WYd}
~
nlt�%d|�zt�&|�Wn
t�y~YnwWYd}
~
q�d}
~
wwd|d<t�d|�'d��t(|d��r�|j)�*|j+|�n|j)�*|j+|�|�,d	dd�}tj�||�}zt�-||�||d<Wq�t�y�Yq�wWd�IdHn1IdH�s�wYWn<tj.j/tf�y&}
z+t!|
tj.j/��st!|
t��r!|
j"t"j#t"j$fv�r!t�%d|
�WYd}
~
WWd|_dS�d}
~
wwWnt0�y9tj1ddd�YnwWd|_dSWd|_dSd|_w)zF
        Async implementation of _process_process_queue_async
        FNcSr�rr�r�rrrr^i	r�z<Minion._process_process_queue_async_impl.<locals>.<listcomp>rr�zPBackpressure active: Pausing queue processing due to recent resource exhaustion.zJBackpressure active: Pausing queue processing due to approaching FD limit.rr�zFProcess queue processing: found %d queued files (Running: %d, Max: %d)�
cSr�rr�r�rrrr^�	szUProcess queue processing: Process count changed, stopping batch. Current: %d, Max: %d�rbzRProcess queue processing: Too many open files, cannot read queued job %s. Waiting.�'Failed to load queued job %s, removing.Tr�zRe-submitting queued job %sr�r�r�r0rzgProcess queue processing: Unable to acquire queue lock due to system resource exhaustion (%s). Waiting.zProcess queue processing failedrb)2rLr?rtr�rIr_r`r�r/rrrzrr�r~r rBrHrr�r�r�rvrwr��sortrrarrr�r�rQrrr!r#r$rCrvr=rir�r=r��replace�renamer�rr�r�)r�r�r
rrrr�Zslots_availabler`Zcurrent_process_countZcurrent_process_count_maxrr|rT�
running_fn�running_pathrrrr3S	s&�
����
�
���
��
�
�
����
����
������������*����
������
�z(Minion._process_process_queue_async_implcCsBt��}|�|jj���|�|jj���|�|jj���|S)zG
        Return a single context manager for the minion's data
        )�
contextlib�	ExitStack�
enter_contextr�Zcontext_dict�cloner�r�)r�Z	exitstackrrr�ctx�	s
z
Minion.ctxc	Cs,|r|tjj_|sI||dd�}||_t|d�s/|j|dd�\}}}}	||_||_||_	|	|_
t|d�sItjjj
|�dd�d�}
t|d	|
d
�|_dd�}tjjj�t�t||d
����0tjjj�|j��||||�Wd�n1swwYWd�dSWd�dS1s�wYdS)NF)rPr�r�)r�rr��r�rl�rYcSs8t|dt�st|dt�rt�|||�St�|||�S)Nr�)rr�tupler�rQ�_thread_multi_return�_thread_return)�minion_instancerIr|rrr�run_func
sz Minion._target.<locals>.run_func)r|rI)rLrrrr�rir�r�r�r�r�r?r��get_uidr=rorr�r��
stack_contextZStackContext�	functools�partialrrA)�clsrGrIr|r�rr�r�r�r�rYrHrrrr
s:

�

���"�zMinion._targetcCs|d}|jr!|jd�dd�r!|jd�dg�}|dkr!||vr!d}|jd�dd�r=|jd�dg�}|dkr=||vr=d}|rCtd��||jvrV|j|}t|||�\}	}
n|}||}	}
d	|jjd
d<t|t�rn|g}nt|t	�ru|s|td�
|���|�d
d�r�|ddkr�d|d<t�d|�|D]'}|�d�}||j
vr�td|�d���|j
|||||	|
�}
|
dur�|
Sq�dS)z�
        Executes a function within a job given it's name, the args and the executors.
        It also checks if the function is allowed to run if 'blackout mode' is enabled.
        Fr�Zminion_blackoutZminion_blackout_whitelistzsaltutil.refresh_pillarTr�z�Minion in blackout mode. Set 'minion_blackout' to False in pillar or grains to resume operations. Only saltutil.refresh_pillar allowed in blackout mode.r�__context__�retcodezDWrong executors specification: {}. String or non-empty list expectedZ	sudo_userr,rZZsudozExecutors list %sz.executez
Executor 'z' is not availableN)r�rIr=rr�r��packrrr�r�rErBrFr�)r��
function_name�
function_argsr�rIr|Zminion_blackout_violationr&r{rqrmr6�fname�return_datarrr�_execute_job_function"
sR�



��

�zMinion._execute_job_functioncs���tj��jt|d��}|�dd�r!tjj	�
|j�d��dt��i}|�
|�t�d|d|d�tjj�|d��}|�tj�|��Wd�n1sSwYd	d
i}|d�|d}|�d
�put�d
g�pu|�d
dg�}	t��fdd�|	D��}
�zt��jvs�|
du�roz����||	||��t�d|d��t�tj�r�d}i}�D]8}
t|
t�r�t|t�r�|�
|
�n	|s�g}|�|
�t|dd|dt|�gd�}d|
i}��||�|d7}q�||d<n�|d<�jj d�dtj!j"j#�}|tj!j"j#k�r/zt�t��rt$�fdd�dD��}nd}Wnt%�y&d}Ynw|�s/tj!j"j&}||d<|tj!j"j#k|d	<W�n|t'�ys}z(d��d�}tj(|dd�|�d |��|d<d!|d"<tj!j"j&|d<WYd}~�nLd}~wt)�y�}z#tj*d#�|t+j,d$�d%|��|d<d!|d"<tj!j"j&|d<WYd}~�nd}~wt-�y�}z%tj*d&�|t+j,d$�d'��d(|��|d<d!|d"<tj!j"j&|d<WYd}~n�d}~wt.�y}z"t�*d&�|�d'��d(|��|d<d!|d"<tj!j"j&|d<WYd}~n�d}~wt/�y8}z+d)�0�|�j�j1�pd*�}tj2|t+j,d$�||d<d!|d"<tj!j"j&|d<WYd}~n�d}~wt%�ynd+}tj2|dd$�tjj*j3tj4�5|�||d,�|�d t6�7���|d<d!|d"<tj!j"j&|d<YnMw�jd-��d.��}|�r��j�8��|�<||d<n#�j�8��|d<��9d/�d}|�j:v�r�|dd0�0�j:|�7<d
|d	<tj!j"j&|d<d!|d"<t|dt��r�|d�d1��r�t�;d2|d�Wzt�<|�WdSt=�y�YdSw|d|d<|d|d<|d|d3<d4|v�r|d4|d4<d5|v�r|d5|d5<d6|v�r)t|d6t��r$|d6|d6<nt�2d7��j>�r>�j?|�j@d8�jAdd9�d:�nt�2d;|d�t|�d�t��rh|d<�rbd=�|d<|df�|d<<n|d|d<<t�(d>|�|d<�r�t|d<t��r�d?|v�r�|d?|d?<d@|v�r�|d@|d@<|d|d<tB|d<�9d=��D]B}z#|�dA�}|�jCv�r��jC||�n
�jC�8|�}t�*dB||�W�q�t%�y�}zt�DdC|d|�WYd}~�q�d}~wwWzt�<|�WdSt=�y�YdSwzt�<|�Wwt=�y	Yww)D�s
        This method should be used as a threading target, start the actual
        minion side execution.
        r�rTz._thread_returnr�z!Starting a new job %s with PID %sr�N�successFr�r�module_executors�direct_callcs0g|]}|�d��jvr�j|�d����qS)z.allow_missing_func)r�)r\�executor)rQrGrrr^z
s��z)Minion._thread_return.<locals>.<listcomp>z*Job %s execution finished, return_data: %sr�progr��job�returnr0rNrOc3s�|]	}��|d�VqdS)TNr�r�)rTrr�	<genexpr>�
s
�
�
�z(Minion._thread_return.<locals>.<genexpr>��resultrWzCommand required for 'z' not foundrbz: �nested�outz#A command in '%s' had a problem: %s)Zexc_info_on_loglevelzERROR: zProblem executing '%s': %szERROR executing 'z': z%Passed invalid arguments to {}: {}
{}r,z'The minion function caused an exception)r\zsys.doc�*r�z Possible reasons: '{}'Z
__no_return__z:Suppressing return for job %s because __no_return__ is set�fun_argsr��	master_id�metadataz6The metadata parameter must be a dictionary. Ignoring.r��rr��3Minion not connected, cannot return data for job %srK�,zminion return: %s�
ret_config�
ret_kwargs�	.returnerz#Returner %s could not be loaded: %s� The return failed for job %s: %s)Er�r_r`rarr�r=rLr?r��appendproctitler
r�rRrBr�rr�writer��dumpsr�r�r�rUrr�types�
GeneratorTypersrvrr�rPr�r��EX_OKr�r�r�rrHrrC�logging�DEBUGrrr�rEr"rDZfire_exceptionr�ZMinionErrorr�r��missing_fun_stringr�r�rFrvrwr��_return_pubrIrrr�r�)rMrGrIr|rn�sdatarrKrRr�Zallow_missing_funcs�indZiretZsingler�Z
event_datarO�func_resultrTr7Zdocs�mod_namer-Zreturner_strZreturner_errr)rQrGrTrrF`
s�
�

����	


��
�


�
���
�����������
�
�� �>��


�
���



���
�������zMinion._thread_returncsh|��tj�|jt|d��}|�dd�r!tjj	�
|j�d��dt��i}|�
|�t�d|d�z#tjj�|d��}|�tj�|��Wd�n1sQwYWn/ty�}z#|jtjtjtjtjfvr�t�d	|d|�t��|_WYd}~dS�d}~ww|�d
d�}t|d�}	|r�dg|	dg|	dg|	d
��niiid
��|�d�p�t|dg�p�|�ddg�}
�zt d|	�D]�}|d|}|d|}
|s�d�d|<ze|�!||
|
||�}|r�|n|d|�|�d�<|j"j#d�dd�}|dk�r0zt$�d�t%��rt&��fdd�dD��}nd}Wnt'�y*d}Ynw|�s0d}|�d�<|dk�d�<Wn1t'�yp}z$t(�)�}t�*d|�|�r\|�d|<n
|�d|d|<WYd}~nd}~ww|d�d<|d�d<|d�d<d|v�r�|d�d<q�d|v�r�|d�d<|j+�r�t�d|d�|�,��nt�*d|d�Wzt�-|�Wnt�y�Ynwzt�-|�Wwt�y�Yww|d �r0d!|v�r�|d!�d!<d"|v�r�|d"�d"<t.|d �/d#��D]4}|d$�d$<z
|j0|�d%���W�q�t'�y/}zt�d&|d|�WYd}~�q�d}~wwdSdS)'rVr�rTz._thread_multi_returnr�zStarting a new job with PID %sr�NzOSystem resource limit reached while writing proc file for job %s: %s. Aborting.�multifunc_orderedFr�)r]rOrWrXrYrrrWr]rNrOc3s$�|]
}�d��|d�VqdS)r]TNr�r��r�rKrrr^�s
��
�z.Minion._thread_multi_return.<locals>.<genexpr>r_r0z+The minion function caused an exception: %srdr�rfz$Attempting to return data for job %srhrKrjrkrir�rlrm)1r�r_r`rarr�r=rLr?r�rnr
r�rRrBr�rrror�rprwr!r"r#r$r%rCr r~r�r�rrUr�rPrrrsr�r�r�r�rDr�rwrvrr�r�)rMrGrIr|rnrxrrTr|Z	num_funcsr�ryrQrRrTrOrzZtrbr-rr}rrETs�
���
���

��
��

�����
�
�����


 ���zMinion._thread_multi_return�_returnc
Csd|�d|�d��}|�d|�d��}tj�|jt|��}tj�|�r3zt�|�Wn	ty2Ynwt	�
d|�|�d�dkrFt	�d|�t	�d	|�|d
kr�||j
d|||�d�|�d
�|�d�|�d�d�}d|vrt|d|d<i|d<|��D]\}}|�d�r�q|||d|<q|n||j
dd�}|��D]\}}|||<q�d|vr�t|dt�r�|d|d<n't	�d|d�nz|j|j}	Wntttfy�Yn
wt|	t�r�|	|d<|j
d�r0|ddkr�tjj�|j
�|d<ztjj�|j
|d|�W|St�y/}
z!|
jtjtjfv�r#t	�d|d|
�t � �|_!n�WYd}
~
|Sd}
~
ww|S)Nr��__jid__r��__fun__�!Returning information for job: %sZcommentzJob queued for executionz&Sending 'Job Queued' return for job %szReturn data: %s�_syndic_returnrYr�tgt�tgt_type�__load__)r�r�r�r�rr�r�r��
__master_id__rer]�__r�)r�r�rb�+Invalid outputter %s. This is likely a bug.�
cache_jobs�reqz6System resource limit reached while caching job %s: %s)"r=r_r`rarr�rurvrwrBr�rDrFrIrtr�rrrCr��
__outputter__r�r�r�rLr?r�Zgen_jidr�r�r!r#r$r r~)r�rK�ret_cmdr�r�rnr�r�r��oputrTrrr�_prepare_return_pub�s���

�
�
��
���
zMinion._prepare_return_pubc#s��|�d|�d���|�||�}|jdsdS�fdd�}z
|j||d�V}Wnty5|�d}Ynwt�d|�tjj	j
�|��)	Nr�r�pub_retr,c�t�d��dS�Nz�The minion failed to return the job information for job %s. This is often due to the master being shut down or overloaded. If the master is running, consider increasing the worker_threads value.T�rBrDr��r�rrr��
�z0Minion._return_pub_main.<locals>.timeout_handlerr��ret_val = %s)r=r�rIr�r
rBrFrLr�r�r�r�)r�rKr�r�r�r��ret_valrr�r�_return_pub_mains�

�zMinion._return_pub_mainc	s�|�d|�d���|�||�}|jdsdS�fdd�}|jdd�|jd	}t||�}z	|j||d
�}Wn
tyB|�YdSwt�d|�|S)�P
        Return the data from the executed command to the master server
        r�rr�r,cr�r�r�r�r�rrr�6r�z+Minion._return_pub.<locals>.timeout_handlerTrgr�r�r�)	r=r�rIrrr�r
rBrF)	r�rKr�r�r�r�Zretry_budgetZeffective_timeoutr�rr�rrw-s 
�
�zMinion._return_pubcs�t|t�s|g}i}|D]�}|�d|�d���|�d|�d��}tj�|jt���}tj�|�rAzt�	|�Wn	t
y@Ynwt�d��|�
�i�}	|dkr�|	sp|	�|jd�||�d�|�d	�|�d
�|�d�id��d
|vrz|d
|	d<|��D]\}
}|
�d�r�q~||	d|
<q~n|	�d|jdi�|��D]\}
}||	|
<q�d|vr�t|dt�r�|d|	d<n't�d|d�nz|j|j}Wntttfy�Yn
wt|t�r�||	d<|jdr�tjj�|j|	d|�q|t|���d�}	�fdd�}
|�rz	|j|	|d�}Wn1t�y|
�YdSwtjj j!�"|
��|j#|	|dd�d�}Wd�n	1�s:wYt�$d|�|S)r�r�rr�r�r�r�r�rr�r�r�)r�r�r�rr�r�r�r]r�rer�r]rbr�r�)r�r�cr�r�r�r�r�rrr��r�z1Minion._return_pub_multi.<locals>.timeout_handlerr�r,cS�dSrr)�frrrr��z*Minion._return_pub_multi.<locals>.<lambda>)r��callbackNr�)%rrr�r=r_r`rarr�rurvrwrBr��
setdefaultrRrIrtr�rCr�r�r�r�r�rLr?r�r��valuesr�r
r�r�rJ�ExceptionStackContextr�rF)r�Zretsr�r��sync�jidsrKr�rnr�r�r�r�r�r�rr�r�_return_pub_multiPs�
���
�
�

�
�
��zMinion._return_pub_multicCs�|jdrb|j�dd�dkr|j�dd�dkrt�d�dSd|j�d	d
�d�}|jddkr<d
|d<|jdg|d<n|jddkrPd|d<|jdg|d<nd|d<g|d<|j�|j|�dSdS)zX
        Execute a state run based on information set in the minion config file
        Zstartup_statesr�r�r�r'r(zpCannot run startup_states when 'master_type' is set to 'disable' and 'file_client' is set to 'remote'. Skipping.r�Z
ext_job_cacher,)r�rKZslsz	state.slsr�Zsls_listr�topz	state.topZtop_filezstate.highstateN)rIr=rBrDr�rlr�r�rrr�
_state_run�s"
��zMinion._state_runcCsNd|j�di�vr%d|jvri|jd<|jd�ddidg|d�i�dSdS)z�
        Create a loop that will fire a pillar refresh to inform a master about a change in the grains of this minion
        :param refresh_interval_in_minutes:
        :return: None
        Z__update_grainsr�z
event.fire�grains_refresh)r�rqr�N)rIr=rR)r�Zrefresh_interval_in_minutesrrr�_refresh_grains_watcher�s


����zMinion._refresh_grains_watcherccsz�d}|jdr
d}|jdr!|jd�|jdt���d|d�V|jd�|jdt���t|jdd	gd
�|d�VdS)NFr�T�enable_legacy_startup_eventszMinion {} started at {}r�Zminion_start)r�r`r�)rIr�rEr �asctimer)r�Zinclude_grainsrrr�_fire_master_minion_start�s�

��z Minion._fire_master_minion_startcCsVt|d�sdSt�d|�|j||d�\|_|_}|_|j|j_|j|j_|��dS)�6
        Refresh the functions and returners.
        r�NzRefreshing modules. Notify=%s)r�)	rirBrHr�r�r�r�r��beacons_refresh)r�r�r�r�rrr�module_refresh�s
�

zMinion.module_refreshcCsf|jsdSt�d�i}t|d�rt|jd�r|jj}t|d�r%|j��tjj|j	|j
|d�|_dS)r�NzRefreshing beacons.r��interval_map)r�)r�rBrHrir�r�Z
close_beaconsrL�BeaconrIr�)r�Zprev_interval_maprrrr��s



�zMinion.beacons_refreshcCst�d�tj�|j�|_dS)z&
        Refresh the matchers
        zRefreshing matchers.N)rBrHrLr�r�rIrrrr�matchers_refresh
s
zMinion.matchers_refreshcCs�t|�D]	}||vr
||=qi}|D]-}||}dd�|D�}||vr?tjjj||||d�}|�d�r;||||<q|||<q|D]}||vrN||||<qB|S)z0
        Refresh the schedule in pillar
        cSsg|]	}|�d�r|�qSr�)r�)r\�itemrrrr^(
sz2Minion.pillar_schedule_refresh.<locals>.<listcomp>)�ignore�new)r�rLr?Z
dictdifferZ	deep_diffr=)r�rr�r�Zpillar_scheduleZ
schedule_itemr��diffrrr�pillar_schedule_refresh
s*�
�
��zMinion.pillar_schedule_refreshc	cs|�|�|�|jrft�d�tjj|j|jd|jd|jd|j�d�|d�}z8z|�	�V}Wnt
y>t�d�Ynw|jd�d	i�}|�d	i�}|�||�|d	<||jd<W|�
�n|�
�w|��|��tjjjd
|jd|jd��9}z|jd
ditjjjd�VWnty�}zt�d|�WYd}~n
d}~wwWd�dSWd�dS1s�wYdS)z$
        Refresh the pillar
        zRefreshing pillar.r�r�r�r�)r��clean_cachezDPillar data could not be refreshed. One or more masters may be down!r�r�r�F�rIr�r�ZcompleteT)r�z.Error firing pillar refresh complete event: %sN)r�r�rBrHrLr�r�rIr=r�rrCr�r,r�r�r?r@rCr�r�r�r�ZMINION_PILLAR_REFRESH_COMPLETEr�)	r�r�r�r�Z
new_pillarZcurrent_scheduleZnew_scheduleZevtrTrrr�pillar_refresh:
s\�


����
�������	"�zMinion.pillar_refreshcCs|�dd�}|�dd�}|�dd�}|�dd�}|�dd�}|�dd�}d||ffd	||ffd
|||ffd|ffd|ffd
||ffd|ffd||ffd||ffd||ffd|ffd|ffdd|ffd||ffd�}	z|	�|�\}
}t|j|
�|�WdSty�t�d|�YdSw)r�r{Nr6r��wherer�r�r�r��
modify_jobZenable_scheduleZdisable_schedule�
enable_job�run_job�disable_job�postpone_job�skip_job�reloadr�)�
save_scheduler�get_next_fire_time�
job_status)�deleter&�modify�enabler�r�r�r�r�r�r�r�r�r�r�z4Function "%s" is unavailable in salt.utils.scheduler)r=r�r�r�rBrC)r�r�r|r{r6r�r�r�r��funcs�alias�paramsrrr�manage_schedulei
s8






��zMinion.manage_schedulec
Cs�|jsdS|�dd�}|�dd�}|�dd�}|�dd�}|�dd�}d||d�fd	||d�fd
d|ifdifdifd
d|ifdd|ifd||d�fdifd||d�fdifd�}z|�|�\}	}
t|j|	�di|
��WdSty~t�d|�YdSw)z 
        Manage Beacons
        Nr{r6�beacon_data�include_pillar�include_optsZ
add_beacon)r6r�Z
modify_beaconZ
delete_beaconZenable_beaconsZdisable_beacons�
enable_beacon�disable_beaconZlist_beacons)r�r�Zlist_available_beacons�validate_beacon�reset)r&r�r�r�r�r�r�r�Zlist_availabler�r�z2Function "%s" is unavailable in salt.utils.beaconsr)r�r=r�r�r�rBrC)r�r�r|r{r6r�r�r�r�r�r�rrr�manage_beacons�
s:


����zMinion.manage_beaconscCsP|�dd�}|durdS|�dd�}|�dd�}ddlmm}|�|||�S)z
        Set the salt-minion main process environment according to
        the data contained in the minion event data
        �environNF�false_unsets�	clear_allr)r=Zsalt.modules.environ�modulesr�Zsetenv)r�r�r|r�r�r�Zmod_environrrr�environ_setenv�
szMinion.environ_setenvc
Cs�|jdur	d|_n$|jdurt�d|jj|jj�dS|jdur-t�d|jj|jj�dSzt�d|jjtjj�	��WdSt
ye}ztjtjj��rPt
jpRt
jd|jj|d�WYd}~dSd}~ww)	z�
        Set the minion running flag and issue the appropriate warnings if
        the minion cannot be started or is already running
        NTFz7This %s was scheduled to stop. Not running %s.tune_in()z4This %s is already running. Not running %s.tune_in()z%s is starting as user '%s'z)Failed to get the user who is starting %srb)ryrBrCr!r
r�rLr?r�Zget_userr�r�Z
is_windowsrtru�ERROR)r�rMrrr�	_pre_tune�
s>

�
�

����zMinion._pre_tunec	Cs�tjjj�|j��3}|j|d<z|j||��|jdd�}|WWd�St	y;t
�d�YWd�dSw1s?wYdS)z.
        Send mine data to the master
        r�r�r�Nz#Unable to send mine data to master.)rLr�r��
ReqChannelr�rIr�r�rr
rBrD)r�r�r|r�rKrrr�
_mine_send�
s 
��	
�	�zMinion._mine_sendccsF	�|jstjjj���tjjj�	|�\}}d|vr7|j
�d�dkr7|d}||jvr1tjjj���|j|}n|}t
�d|j
d|�|�d�rY|j|�dd�|�d	d�d
�dS|�d��r9|�dd
�dd�\}}||j
dk�r-d}z|jj||��|j
dd�V}Wn\tjjy�t
�d||�tjjjd|j
d|jd��0}	z|	�ddd�d|���VWnty�}
zt
�d|
�WYd}
~
nd}
~
wwWd�n1s�wYtjjj���wtjjjd|j
d|jd��0}	z|	�d|id|���VWnt�y}
zt
�d|
�WYd}
~
nd}
~
wwWd�n1�s$wYdSdSt
�d|d||�dS|�d��rQ|j|�dd�|�dd�d�VdS|�d ��r]|��dS|�d!��ri|��dS|�d"��rw|�||�dS|�d#��r�|� ||�dS|�d$��r�|�dd��s�|j!|j
d%k�r�|jd&d'�V|j
d%|_!dSdS|�d(��r�|�"||�dS|�d)��r�|�#||�dS|�d*��r�|j$�r�t
�d+|d,|j
d�|�%|d-|d,|d.|d/�VdSt
�d0|j
d|d,�dS|�t&d1d2���s|�t&d3d2���r�|�t&d1d2���r+|d|j
dk�r+tjjj���|�t&d3d2���rb|d|j
d4d5k�rMt'd6�(|d|j
d���|d|j
dd5k�rbt'd7�(|d���|j$�r�d|_$t
�)d8|j
d�|j
d9d:k�r�|j*j+t&d;d2�d<�t
�)d=�t,|d>��r�|j-�.d�t,|j-d?��r�|j-j/�0�t,|j-d@��r�|j-�1�t,|dA��r�|j�r�|j�2�Vtjjj�3d5�Vd|_z|j4|j
d&|�t&d3d2��dB�V\}|_-Wn
t5�y�Ynw|j$�r�||j
d<t
�)dC|j
d�tj6j7j8j9|j
|jdD�|_|j�:�V|j*�;dE�|j
dE<|�<�\|_=|_>|_?|_@|j=|j*_=|j-�.|jA�|�B�Vt
�)dF�|j
d9d:k�r�dG|j
dHd&dd|j
dd&dI�dJ�}|j*jCt&d;|j
ddK�|dL�|j
dM�r�d4|j
v�r�|j
d|j
d4d5k�r�dN|j
dOd&ddd|j
d4d5idJ�}|j*jCt&d3d2�|dL�dS|j*j+t&d3d2�d&dP�dSdSdSdSd&|_D|j�E�dSdS|�t&dQd2���r
|j$�s|j
dRdSk�r	t
�)dT|j
d�d&|_$|j
d9d:k�rdG|j
dHd&dd|j
dd&dI�dJ�}|j*jCt&d;|j
ddK�|dL�dSdSdSdS|�dU��r?|dE�t&d;dVdK���r5|dW�r5t
�dX|dE�Ft&d;dVdK��d�|jG|dYdZ�VdS|�d[��rZ|j$�rXt
�d\|�|�%||�VdSdS|�d]��r�tH|d^�}
t
�d_|
tjIjJjK�|
�|d`�|d`tjIjJjKtH|d^�<dS|�da��r�|j$�r�t
�db�|j%|dcdd�VdSdSdS)ezO
        Handle an event from the epull_sock (all local minion events)
        �proxy_target�	metaproxyZ
deltaproxyz)Minion of '%s' is handling event tag '%s'r+r�r�Fr�)r�r�Z__master_req_channel_payloadr�rsr0Nr�r�z3Timeout encountered while sending %r request. id=%sr�r�r�)rKrCr�z-Error firing master request timeout event: %srKz,Error firing master request return event: %sz5Skipping req for other master: cmd=%s master=%s id=%sr�r�r�)r�r�r�r�r�r�r�r�T)r�r�Z_minion_mineZfire_masterz Forwarding event %s to master %sr�r|r�r�z-Master %s is not connected, dropping event %sr�r�r�r�rz*Bad master '{}' when mine failback is '{}'zAlready connected to '{}'zConnection to master %s lostr�r�r�r5z1Trying to tune in to next master from master-listr�r�r�r})rIr�r�z,Re-initialising subsystems for new master %srBr��$Minion is ready to receive requests!r�r�r�r�r�)r6r�r�r�r�)r6r�r�r�r�z&Connection to master %s re-establishedZ__schedule_returnr,r]zConnected to master %sr~)r�Z_salt_errorz"Forwarding salt error event tag=%szsalt/auth/credsr�z#Updating auth data for %s: %s -> %sZcreds�__beacons_returnzFiring beacons to masterr�)r�)Lr{rLr�r�r�r�r?r@Z	SaltEvent�unpackrIr=Zdeltaproxy_objsrBrHr�r��rsplitr}r�rr�r
rCrCr�r�r�r�r�r�r�r��grains_cacher�r�r�r�r�r
rEr�r�r�rir�r�r�r�r�r�r"r�rr�r�r�r�r��optionr�r�r�r�r��_handle_payloadr�r�rkrpr�r�rDrrr)r�rIr�r|r�Z_minionr�Z
job_masterrKr@rTr+r�r�rrrrF�
s
�




����
��������
����&��

�
���
�������

��
�
��


����
�

���c�	
��
����
�zMinion.handle_eventcCs*t��|j��|jr|j��dSdS)z<
        Clean up subprocesses and spawned threads.
        N)rZactive_childrenrzr�r��cleanup_subprocessesrrrrr�s

�zMinion.cleanup_subprocessescCs�|jsK|��\|_|_|_|_|��|_tj	�
|j�|_
|jr)tj
�|j|j�|_
tjjj|j�dd�d�}t|jd|d�|_|jd|_d|_dSdS)za
        Set up the core minion attributes.
        This is safe to call multiple times.
        r�NrBrlrCr�T)r{r�r�r�r�r�r�r�rLr�r�rIr�r�r�r?r�rIr=rorr�)r�rYrrrr�%s �

�zMinion._setup_corecsd�jsdS����jd}d�jvr0tj��j�j��_�fdd�}|r(|���d|�dSdS)zR
        Set up the beacons.
        This is safe to call multiple times.
        Nr�r�cs�d}z���j�}Wntytjddd�Ynw|rCtjjjd�j	dd��}|�
d|id�Wd�dS1s<wYdSdS)	NzThe beacon errored: Trbr�Fr�r�r�)r�r�r�rBr�rLr?r@rCrIr�)r�r@rrr�handle_beaconsJs ��"��z,Minion.setup_beacons.<locals>.handle_beacons)	r�r�rIr|rLr�r�r�r+)r�r^r�r�rrrrd<s

�zMinion.setup_beaconsc
s�����jd�d�jvryd�jvri�jd<t�d�s2tjjj�j�j�j	�jt
dd�gd��_z�jdrKt�d�jd���
t�jd��Wntyd}zt�d|�WYd	}~nd	}~ww��fd
d�}|rq|���d|�d	Sd	S)zT
        Set up the scheduler.
        This is safe to call multiple times.
        r�r�r�r�)r?r�Zgrains_refresh_everyz;Enabling the grains refresher. Will run every %d minute(s).z[Exception occurred in attempt to initialize grain refresh routine during minion tune-in: %sNcs�����dSr)r�r�r�r�rr�handle_schedule�sz/Minion.setup_scheduler.<locals>.handle_schedule)r�rIr|rirLr?r�r�r�r�r�rBrHr��absr�rCr+)r�r^rTr�rr�rre_sB





�
������zMinion.setup_schedulercCr()za
        Set up the state queue processing.
        This is safe to call multiple times.
        Zstate_queuer�r)N)r|r+�process_state_queuerrrr�setup_state_queue_processing�s


��z#Minion.setup_state_queue_processingcCr.)z�
        Check the state queue for pending jobs and execute them if safe.
        Runs as an async task on the main loop.
        �_state_queue_processing_activeFNT)r�r�r�r=�_process_state_queue_asyncrrrrr��r1zMinion.process_state_queueccr2)z4
        Async body of process_state_queue.
        N)�_process_state_queue_async_implrrrrr��r4z!Minion._process_state_queue_asyncc�sl��z��z�tjj�|j�}tj�|�sWWd|_dS�z;tjj�	|j�4IdH�� g}zt�
|�D]}|�d�rC|�d�rC|�
|�q2Wn	tyNYnw|sc	Wd�IdHWWWd|_dSdd�}|j|d�|d}d}z |�d�}t|�d	kr�|d
}|�d�r�|dd�}t|�}Wnttfy�Ynwt�dt|�||�z
tjj�|j�}	Wn(ty�}
zt�d
|
�WYd}
~
Wd�IdHWWWd|_dSd}
~
wwtjj�|j||	�}|�rt�d|dd�|D��	Wd�IdHWWWd|_dS|d}tj�||�}z!tjj�|d��}
tj�|
�}Wd�n	1�s-wYWnittf�y�}
zZt |
t��rj|
j!t!j"t!j#fv�rjt�d|�WYd}
~
Wd�IdHWWWd|_dSt�d|�zt�$|�Wn
t�y�YnwWYd}
~
Wd�IdHWWWd|_dSd}
~
wwz,|�d�}t|�d	k�r�|d
}|�d��r�|dd�}t|�||d<t�%d|�Wnttf�y�Ynwt�%d|�&d��|�'ddd�}tj�||�}zt�(||�||d<Wn*t�y#}
zt�d||
�WYd}
~
Wd�IdHWWWd|_dSd}
~
wwd|d<t)|d��r7|j*�+|j,|�n|j*�+|j,|�Wd�IdHn1IdH�sPwYWn<ttj-j.f�y�}
z+t |
tj-j.��s{t |
t��r�|
j!t!j"t!j#fv�r�t�d|
�WYd}
~
WWd|_dS�d}
~
wwWnt/�y�tj0ddd �YnwWd|_dSWd|_dSd|_w)!NFrr�c	Ssfz$|�d�}t|�dkr |d}|�d�r|dd�}t|�WSWtd�Sttfy2Ytd�Sw)Nr�r�rsr�����inf)r�r�r�rArQ�
IndexError�float)r��parts�jid_strrrr�sort_key�s


�	��z8Minion._process_state_queue_async_impl.<locals>.sort_key)r�rr�r�rsr�zBState queue processing: found %d queued files in %s, candidate: %sz]State queue processing: System resource exhaustion preventing active job check (%s). Waiting.zPState queue processing: candidate job %s is blocked by active jobs: %s. Waiting.cSsg|]}|d�qSr�r)r\�jrrrr^�sz:Minion._process_state_queue_async_impl.<locals>.<listcomp>r6zPState queue processing: Too many open files, cannot read queued job %s. Waiting.r7r�z2Updated execution JID to match queued filename: %sz2Popping queued job %s from the queue for executionr�r0rz'Failed to rename queued job file %s: %sTr�r�zeState queue processing: Unable to acquire queue lock due to system resource exhaustion (%s). Waiting.zState queue processing failedrb)1rLr?rtr�rIr_r`r�r�rr�r�r�rvrwr8r�r�rArQr�rBrFZget_active_statesrCZcheck_prior_running_statesrHrarrr�r�rrr!r#r$rvr�r=r9r:rir�r=r�r�rr�r�)r�r�rr�r�Zcandidate_fnZ
candidate_jidr�r�ZactiverTZ
blocking_jobsr`rr|r;r<rrrr��sT�
,��
����
(��

���	�
�
(�����
(�����
�
(��
�
(��
����
��
(����*����
����
��
�z&Minion._process_state_queue_async_implr0cCs<||jvrdStjjj�||d�|j|<|j|��dS)z�
        Add a periodic callback to the event loop and call its start method.
        If a callback by the given name exists this method returns False
        FrT)r|rLr�r�r�PeriodicCallbackr`)r�r6�methodr*rrrr+[s


�zMinion.add_periodic_callbackcCs&|j�|d�}|dur
dS|��dS)z~
        Remove a periodic callback.
        If a callback by the given name does not exist this method returns False
        NFT)r|rbrp)r�r6r�rrr�remove_periodic_callbackis
zMinion.remove_periodic_callbackc	s����t�d�jd�|r-�j�dd�r�jdd��j�dd�r)�jdd�����jr<�j	�
�j�t�d�t
�trGtjj�������������������d	�j��j�d
d�d}|dkr��jr��fd
d�}��d���d||�t�d�r��jdur��j��j�n
�j�d�dkr�t�d��jdtjdd�|r�z�j	� ��j!r���"�WdSWdSt#t$fy���"�YdSwdS)�k
        Lock onto the publisher. This is the main event loop for the minion
        :rtype : None
        zMinion '%s' trying to tune inr�r\FTr]r_r�r��
ping_intervalrr�csNz�fdd�}�jj�jdd|d�WdSty&tjdtjd�YdSw)Ncsl�j�dd�r4t�d��j�dd�}t�d|�z�jdt��WdSty3t�d�YdSwdS)	N�
auth_safemodeFz5** Master Ping failed. Attempting to restart minion**Zrandom_reauth_delayr�z delaying random_reauth_delay %sszservice.restartz�ping_interval reached without response from the master, but service.restart could not be run to restart the minion daemon. ping_interval requires that the minion is running under an init system.)	rIr=rBrCr�r�r�r�rD)r��delayrrr�ping_timeout_handler�s�
���zAMinion.tune_in.<locals>.ping_master.<locals>.ping_timeout_handler�pingZminion_ping)r�zAttempt to ping master failed.)Zexc_on_loglevel)r�rlr�r�rBrDrtru)r�rrr�ping_master�s���z#Minion.tune_in.<locals>.ping_masterr�r�Nr�r�z;No connection to master found. Scheduled jobs will not run.Z
gc_collectr))%r�rBrHrIr=rdrer�r�r�rlr�r�r�HAS_WIN_FUNCTIONSrLr?Z
win_functionsZenable_ctrl_logoff_handlerr�r�r-r+r�r�rir�r�r�rC�gcZcollectr`rkr,r��RuntimeError)r�r`r�r�rrrrgusV

!
	
�
���zMinion.tune_incCsd|dur,|ddkr.|�|d�r|j�|j|d�dS|jdr0t�d|d�dSdSdSdS)N�enc�aesr�Z
zmq_filteringz8Broadcast message received not for this minion, Load: %s)�_target_loadr�r=r�rIrBrF�r�r�rrrr��s
�
��zMinion._handle_payloadcCs�d|vsd|vsd|vsd|vrdSd|vrJ|j�d�|d�d�}|dur(dS|ddvr@|�d	t�}||d|d
�s>dSdS||d�sHdSdS|jd|d�sUdSdS)
Nr�r�r�rFr�z{}_match.match)ZgrainZ
grain_pcrer��	delimiter)rzglob_match.matchT)r�r=rEr)r�r�Z
match_funcrrrrr�s,	���zMinion._target_loadccs��d|_t|d�r|jdur|j��|j��t|d�r|`t|d�r7|jdur7|j�d�|j��d|_t|d�rS|j	durS|j	�
�Vtjj
j�d�Vd|_	t|d�rd|j��D]}|��q]dSdS)	z�
        Async teardown for use on the minion I/O loop. Ensures REQ transport
        ``close_async`` completes before the next connect attempt (see ``connect_master``).
        Fr<Nr�r�r}rr|)ryrir<rJrKr�r�r�r�r}r�rLr�r�r�r"r|r�rp�r��cbrrrrh
s(�





�zMinion.destroy_asynccCsd|_t|d�r|jdur|j��|j��t|d�r|`t|d�r3|jdur3|j�d�|j��t|d�rB|j	durB|j	��t|d�rS|j
��D]}|��qLt|d�rk|j
durkt|j
d	�rh|j
��i|_
t|d
�r�|jdur�t|jd	�r�|j��i|_t|d�r�|jdur�t|jd	�r�|j��i|_t|d�r�|jdur�t|jd	�r�|j��i|_t|d
�r�|jdur�t|jd	�r�|j��i|_t|d�r�|jdur�t|jd	�r�|j��i|_t|d�r�|jdur�t|jd	�r�|j��i|_dSdSdS)z&
        Tear down the minion
        Fr<Nr�r�r}r|r�r,r�r�r�r�r�r?)ryrir<rJrKr�r�r�r�r}r|r�rpr�r,r�r�r�r�r�r?rrrrr,%sV














�zMinion.destroycCr/rr0rrrrrAWr2zMinion.__del__)r�TNNNT�NFr	)FFNNN)NNNNr�NF)r~)r~r�)r~r�T)FF)r0�T)Lr
rrr"r�r�rxr�rLr�r�r�rrfr�r�r�r�r�r�r�r�r�r�r�r�rrrrr	r-r,r0r3rA�classmethodrrUrFrEr�r�rwr�r�r�r�r�r�r�r�r�r�r�r�r�r�rFr�r�rdrer�r�r�r�r+r�rgr�rrhr,rAr#rrr rrQ�s��r
&
 



s
�V




�
�
�


6 I	

$

!>
t

{
D

#]



!.$(&

!


#-


2
e
#
2rQcs�eZdZdZ�fdd�Zdd�Zdd�Zdd	�Zej	j
jjd
d��Z
dd
�Zdd�Zdd�Zej	j
jjdd��Zej	j
jjdd��Z�fdd�Z�ZS)�Syndicz�
    Make a Syndic minion, this minion will use the minion keys on the
    master to authenticate with a higher level master.
    csnd|_d|_|�d�|_d|_d|d<d|d<t�j|fi|��tj�	|�|_
t�|_i|_
g|_d|_dS)NZ	interfaceTr�r0r�)r)�forward_eventsr=Z_syndic_interfaceZ_syndicrr�rLr�r$�mminionr�jid_forward_cacher��
raw_eventsZ
pub_future)r�rIrmr rrr�cs
zSyndic.__init__cCsJt|�d|jd��d|d<|�dd�|j�dd�kr#|�|�dSdS)r��tor�r0rerN)rAr=rI�
syndic_cmdr�rrrr�rs�zSyndic._handle_decoded_payloadc
Cs�d|vrd|d<d|�dg�i}dD]}||vr||||<qdd�}tjjj�|��-|jj|d|d|d	|d|d
|d|df|jd
d�d�|��Wd�dS1s[wYdS)zM
        Take the now clear load and forward it on to the client cmd
        r��globZ	auth_list)rer�cWst�d|d�dS)NzUnable to forward pub data: %sr0Tr�)rqrrrr��sz*Syndic.syndic_cmd.<locals>.timeout_handlerr�r�rrKr�rcSr�rrr�rrrr�r�z#Syndic.syndic_cmd.<locals>.<lambda>)r�r�N)	rbrLr�r�rJr�r)Z	pub_asyncr�)r�r|rm�fieldr�rrrr}s0���
�"�zSyndic.syndic_cmdcCs|jj|||jdd�S�Nr�r�)r}r�rI)r�r�r�rrrr��s�zSyndic._send_req_syncccs �|jj|||jdd�V}|Sr)�async_req_channelr�rIr�rrrr��s
��zSyndic._send_req_asynccCsZ|jdr|�d�|jdt���d�|�d�|jdt���t|jddgd��dS)Nr�zSyndic {} started at {}r�Zsyndic_startr`r2)rIr�rEr r�rrrrr�fire_master_syndic_start�s
��zSyndic.fire_master_syndic_startcCsTtjj|jd|jd�|_|j�|j�tj	jj
�|j�|_tj	jj
�|j�|_dS)z�
        Executes the tune_in sequence but omits extra logging and the
        management of the event bus assuming that these are handled outside
        the tune_in sequence
        �_minion_conf_filerBN)rLr��get_local_clientrIr�r)r�r��_process_cmd_socketr�r�r�r}r�rrrrr�tune_in_no_block�s�zSyndic.tune_in_no_blockcCs8|dur|ddkrt�d�|�|d�dSdSdS)NrrzHandling payloadr�)rBrFr�rrrrr�s
�zSyndic._process_cmd_socketccs��t|d�r|j�d�t|jd�r|j��|`|j|jd�V\}|_|jr8||jd<|j�|j�t�	d�t
jjj
�|��)Nr�r�r�r+r�)rir�r�r�r�rIr�rrBr�rLr�r�r�r�)r�r+rrr�	reconnect�s�



zSyndic.reconnectccsJ�t�|�V|jdur|j��d|_|jdur#|j��d|_dSdS)zT
        Async teardown on the I/O loop (see :meth:`Minion.destroy_async`).
        N)rQrhr)r,rrprrrrrh�s�




�zSyndic.destroy_asynccsjt���|jdur|j��d|_t|d�r$|jdur$|j��d|_|jdur3|j��d|_dSdS)z-
        Tear down the syndic minion
        Nr)rr,r)rirrrprr rrr,�s






�zSyndic.destroy)r
rrr"r�r�rr�rLr�r�r�rr�rrrrrhr,r#rrr rr
]s $






r
cs�eZdZdZdZdZd�fdd�	Zdd�Zej	j
jjdd	��Z
d
d�Zd d
d�Zddd�Zddd�Zdd�Zdd�Zdd�Zdd�Zdd�Zdd�Z�ZS)!�
SyndicManagera�
    Make a MultiMaster syndic minion, this minion will handle relaying jobs and returns from
    all minions connected to it to the list of masters it is connected to.

    Modes (controlled by `syndic_mode`:
        sync: This mode will synchronize all events and publishes from higher level masters
        cluster: This mode will only sync job publishes and returns

    Note: jobs will be returned best-effort to the requesting master. This also means
    (since we are using zmq) that if a job was fired and the master disconnects
    between the publish and return, that the return will end up in a zmq buffer
    in this Syndic headed to that original master.

    In addition, since these classes all seem to use a mix of blocking and non-blocking
    calls (with varying timeouts along the way) this daemon does not handle failure well,
    it will (under most circumstances) stall the daemon for ~15s trying to forward events
    to the down master
    r�Ncs�d|d<t��|�d|_tj�|�|_|j�dd�|_	|j�dd�|_
|jd|_|jd	|_t
��|_t�|_|durGtjjjj��|_n||_g|_i|_g|_t�t�|_i|_dS)
Nr0r�F�syndic_moder��syndic_failoverr�r�r4) rr��_closingrLr�r$rrIr=rr r8r9r�EventZ_has_masterrrr�r�rrrr�r�job_rets�delayed�collections�defaultdictrAr��pub_futures)r�rIr�r rrr� s$

zSyndicManager.__init__cCsTt�|_|jd}t|t�s|g}|D]}t�|j�}||d<|�|�|j|<qdS)zI
        Spawn all the coroutines which will sign in the syndics
        r+N)r�_syndicsrIrrr�r��_connect_syndic)r�rr+rZrrr�_spawn_syndics@s

�zSyndicManager._spawn_syndicsc
cs�d}|d}d}	t�d|d�z#t||jd|jd�}|j|d�V|��|��t�d	|d�WnMt	yd}z#d}t�
d
|d�||jkrQ||j7}t
jjj�|�VWYd}~n"d}~wttfym�ty�d}tjd|ddd
�Ynwq
t
jjj�|��)zL
        Create a syndic, and asynchronously connect it to a master
        rr�FTz"Syndic attempting to connect to %sr+)r�r�r�rz#Syndic successfully connected to %szPError while bringing up syndic for multi-syndic. Is the master at %s responding?Nrarb)rBrHr
�SYNDIC_CONNECT_TIMEOUTr�rfrrr�rrCr9r8rLr�r�r�r"r��
SystemExitr�r�r�)r�rIrir8r�r2rTrrrr)MsP���

�
���&zSyndicManager._connect_syndiccCs>|j|��r|j|��}|��|j|<dSt�d|�dS)zL
        Mark a master as dead. This will start the sign-in routine
        zAAttempting to mark %s as dead, although it is already marked deadN)r(r�r`rrBr�)r�r+r2rrr�_mark_master_dead}s�zSyndicManager._mark_master_deadrc	Cs�|duri}d}|�|�D]9\}}|��r|��r!t�d||�q
zt|��|�|i|��d}Wq
tyFt�d||�|�|�Yq
w|sQt�	d|�dSdS)zd
        Wrapper to call a given func on a syndic, best effort to get the one you asked for
        NF�5Unable to call %s on %s, that syndic is not connectedT�*Unable to call %s on %s, trying another...z!Unable to call %s on any masters!)
�iter_master_optionsr�r�rBrCr�r`rr-r�)r�r{rqrmreZ
successfulr+�
syndic_futurerrr�_call_syndic�s*���zSyndicManager._call_syndiccCsd}|�|�D]�\}}|��r|��rt�d||�q|j�|d�\}}|durt|��s4||kr3dSq|��rnt�d||�|�|�|j|=|j|d7<|j||j	�dd	�krd|j
�|�n	t�
t�|_d
Sqt�
t�|_t|��|�|d|��dd�}||f|j|<d
SdS)
zl
        Wrapper to call the '_return_pub_multi' a syndic, best effort to get the one you asked for
        r�r.r�NFr/r0Zsyndic_retriesr�Tr�)r�r�)r0r�r�rBrCr'r=r-r�rIr$�extendr%r&rAr�r`r)r�r�rer{r+r1r�r|rrr�_return_pub_syndic�sD��
�z SyndicManager._return_pub_syndicccsn�t|j���}|jddkrt�|�||jvr|�d�}n|�|�	||j|fV|s1dS|�d�}q%)zA
        Iterate (in order) over your options for master
        r r�rTN)r�r(ryrIr�r�rbrv)r�rerrrrr0�s�



�z!SyndicManager.iter_master_optionscCsi|_g|_dSr)r#rrrrr�_reset_event_aggregation�s
z&SyndicManager._reset_event_aggregationcCs$|jj�|j�}|j�||j�dSr)r)r@rE�_process_eventr��
add_future�reconnect_event_bus)r�Z	somethingr�rrrr8�sz!SyndicManager.reconnect_event_buscCs�|��tjj|jd|jd�|_|jj�d�t	�
d|jd�i|_g|_|�
�|jj�|j�}|j�||j�tjjj�|j|jdd�|_|j��t�|j��dS)	zU
        Lock onto the publisher. This is the main event loop for the syndic
        rrBr,z$SyndicManager '%s' trying to tune inr�Zsyndic_event_forward_timeoutrN)r*rLr�rrIr�r)r@rDrBrHr#rr5rEr6r7r8r�r�rr��_forward_eventsrr`r)r�r�rrrrg�s$�
�
zSyndicManager.tune_inc
Cs |jj�|�\}}t�d|�d}d}|�d�}t|�dkr/|ddkr/tjj	�
|d�r/d}|jd	krC|�d
d�|j
�d
d�krCd}|r�|dd
kr�d|vr�d|vrUdS|s^t�d�dS|�d
�}|j�|i��|i�}|s�|�d�|d<|d|d<i|d<d�|j
d�}	|d|jvr�|d�|jj|	|d��|j�|d�t|j�|j
dkr�tt|j��}
|
�d�t|
�|_|dur�||d<i}dD]}||vr�||||<q�|||d<dS|jdks�|�r
tjj	�
|��rd|v�rd|v�r|j�||d��dSdSdSdSdS)NzGot event %sFTr��r0r\rsZclusterrerr�rKr]r�z7Return received with matching master_id, not forwardingr�r�rr�z{}.get_loadZmaster_job_cacheZsyndic_jid_forward_cache_hwmr�)r]rOrWrdr�r�r:rO)r|r�)r)r@r�rBrFr�r�rLr?r�Zis_jidrr=rIrHr#r�rErrRrr�r&�sortedr�rbrrrv)
r��rawZmtagr|Z	job_eventZreturn_eventZ	tag_partsr+ZjdictZfstr�tmprKr�rrrr6sn
��

�

�
��

��zSyndicManager._process_eventcCs�t�d�|jr"|j}g|_|jd|t|jddd�|��d�d�|jr0|�|j�}|r0g|_t	|j
���D]}t	|j
|���}|j||d�}|rO|j
|=q7dS)	NzForwarding eventsr�r�r2)r)r�r�r�)rm)re)
rBrFrr2rrIrr$r4r�r#ryr�)r�r��resr+r�rrrr9[s,
����zSyndicManager._forward_eventscCs^|jdurdSd|_|jdur|j��d|_t|d�r+|jdur-|j��d|_dSdSdS)NTr)r!r)r,rirrrrrr,rs




�zSyndicManager.destroyr)rNN)r
rrr"r+ZSYNDIC_EVENT_TIMEOUTr�r*rLr�r�r�rr)r-r2r4r0r5r8rgr6r9r,r#rrr rrs$ 

/


1 Irc@s"eZdZdZ				ddd�ZdS)�ProxyMinionManagerzA
    Create the multi-minion interface but for proxy minions
    NTc	Cst||||||dd�S)rMTrN)�ProxyMinionrRrrrrS�rTz(ProxyMinionManager._create_minion_objectrq)r
rrr"rSrrrrr?~s	�r?cCs<|�d�}||jvr|�|�}d|�d|�d�Sd|�d�S)u]
    Build the error message a proxy minion should abort with when its
    proxymodule's ``init`` and/or ``shutdown`` are unavailable.

    Historically all three call sites (``salt.metaproxy.proxy``,
    ``salt.metaproxy.deltaproxy`` and ``salt.minion.ProxyMinion``) emitted
    a single hard-coded sentence — "Proxymodule X is missing an init() or
    a shutdown() or both" — regardless of *why* those functions weren't
    in the loader. In practice that wording is wrong far more often than
    it is right: a missing dependency, an ``ImportError`` while loading
    the proxymodule, or any ``__virtual__()`` returning ``False`` all
    surface the same misleading message even though the module itself
    defines both functions.

    This helper inspects the proxy ``LazyLoader`` and distinguishes the
    two situations:

    * If the proxymodule itself never loaded (it is in the loader's
      ``missing_modules``), surface the underlying reason via
      :meth:`LazyLoader.missing_fun_string` so the operator can see the
      real cause (e.g. a failed ``__virtual__`` or import error).
    * Otherwise keep the historical "missing an init() or a shutdown()"
      wording — that case really does indicate a malformed proxymodule.

    Both branches end with the same "Salt-proxy aborted." suffix that
    callers and external log scrapers may rely on.
    �.initzProxymodule z could not be loaded: z. Salt-proxy aborted.z[ is missing an init() or a shutdown() or both. Check your proxymodule.  Salt-proxy aborted.)Zloaded_modulesrv)Zproxy_loader�fq_proxynameZproxy_init_func_name�reasonrrr�proxy_load_failure_message�s


�
�rDcCsnd�|dtjjj�}tjj||d�}z|d}Wnty,d}t�d|d�Ynw|d|}||S)Nz{}.{}r�)rOr�r�zMNo metaproxy key found in opts for id %s. Defaulting to standard proxy minionr�)	rErLr�ZlazyZLOADED_BASE_NAMEr�r�rBrH)rIZfn_namerOr�Zmetaproxy_nameZmetaproxy_fnrrr�_metaproxy_call�s��rEc@s�eZdZdZejjjjdd��Z	ejjjjdd��Z
ddd�Zd	d
�Zdd�Z
ejjjjd
d��Zedd��Zedd��Zedd��ZdS)r@z�
    This class instantiates a 'proxy' minion--a minion that does not manipulate
    the host it runs on, but instead manipulates a device that cannot run a minion.
    cC�t|jd�}|||�S)a�
        Function to finish init after connecting to a master

        This is primarily loading modules, pillars, etc. (since they need
        to know which master they connected to)

        If this function is changed, please check Minion._post_master_init
        to see if those changes need to be propagated.

        ProxyMinions need a significantly different post master setup,
        which is why the differences are not factored out into separate helper
        functions.
        Zpost_master_init�rErI)r�r+�mp_callrrrr��s
zProxyMinion._post_master_initcCst|jd�}||||�S)zT
        Function to finish init for the sub proxies

        :rtype : None
        �subproxy_post_master_initrG)r�Z	minion_idrYrHrrrrI�sz%ProxyMinion.subproxy_post_master_initTcCrF)r�rgrG)r�r`rHrrrrg�s
zProxyMinion.tune_incCrF)zQ
        Verify that the publication is valid and applies to this minion
        Ztarget_loadrG)r�r�rHrrrrs
zProxyMinion._target_loadcCrF)Nr�rG)r�r�rHrrrr�s
zProxyMinion._handle_payloadcCrF)NZhandle_decoded_payloadrG)r�r|rHrrrr�s
z#ProxyMinion._handle_decoded_payloadcCst|d�}|||||||�S)Nr�rE)rMrGrIr|r�rrHrrrrs
zProxyMinion._targetcC�t|d�}|||||�S)NZ
thread_returnrJ�rMrGrIr|rHrrrrF�
zProxyMinion._thread_returncCrK)NZthread_multi_returnrJrLrrrrErMz ProxyMinion._thread_multi_returnNr)r
rrr"rLr�r�r�rr�rIrgrr�r�rrrFrErrrrr@�s"




	



r@c@seZdZdZddd�ZdS)�SProxyMiniona
    Create an object that has loaded all of the minion module functions,
    grains, modules, returners etc.  The SProxyMinion allows developers to
    generate all of the salt minion functions and present them with these
    functions for general use.
    FNcCs�tjj�|jd�tj�|j�|jd<tjj|j|jd|jd|jd|j�	d�d��
�|jd<d|jdvrWd|jvrWdj|jdd	�}t�
|�d
|_ttjjj|d��d|jvrf|jdd|jd<tj�|j�|_tjj|j|j|d�|_tjj|j|jd
|j|d
�|_tjj|j|j|j|d�|_tj�|j�|_|j|jd<tjj|j|j|j|d�|_|jdd}|jd|jdd�|j|jjd<|j|jjd<|j|jjd<|jd|jjd<tjj|j|j|d�|_|j|jjd<|j��|�d�|jv�s
|�d�|jv�r$t|j|�}t�
|�d
|_ttjjj|d��|j�	|�d�dd���|_|j|d}||j�tjj|j|jd�|jd<|jddd�|jd|_d |_d!S)"r�r�r�r�r�)r�r�r�r�z�No "proxy" configuration key found in pillar or opts dictionaries for id {id}. Check your pillar/options configuration and contents. Salt-proxy aborted.)r�Fr5r�r�r�r�Z	proxytypezsaltutil.sync_all)r�Z	__proxy__Z__salt__Z__ret__Z
__pillar__Z	__utils__rAz	.shutdownz.module_executorscSsgSrrrrrrr~r�z*SProxyMinion.gen_modules.<locals>.<lambda>)r�zsaltutil.sync_grainsrTN) rLr?Zextmodsr�rIr�r�r�r�r=r�rErBrCryrr�r�r�r�r�r�r�r�r�r�rPZreload_modulesrDrXr�r{)r�r�r��errmsgrBZ
proxy_init_fnrrrr�)s|
����

���

�

zSProxyMinion.gen_modulesr)r
rrr"r�rrrrrN!srN)rrr
r){r"r�r%r=r�r!rKrrtrr_r�rrer�rr r�rqr�rrLZsalt.beaconsZsalt.channel.clientZsalt.cli.daemonsZsalt.clientZ
salt.cryptZsalt.defaults.eventsZsalt.defaults.exitcodesZsalt.enginesZsalt.ext.tornadoZsalt.ext.tornado.genZsalt.ext.tornado.iolooprZsalt.loader.lazyZsalt.payloadZsalt.pillarZsalt.serializers.msgpackZ
salt.syspathsZsalt.transportZsalt.utils.argsZsalt.utils.atomicfileZsalt.utils.contextZsalt.utils.dataZsalt.utils.dictdifferZsalt.utils.dictupdateZsalt.utils.errorZsalt.utils.eventZsalt.utils.extmodsZsalt.utils.filesZsalt.utils.jidZsalt.utils.minionZsalt.utils.minionsr>Zsalt.utils.platformZsalt.utils.processZsalt.utils.scheduleZsalt.utils.ssdpZsalt.utils.stateZsalt.utils.userZsalt.utils.zeromqZsalt._compatrZsalt.configrZ
salt.defaultsrZsalt.exceptionsrrrr	r
rrr
rZ
salt.templaterZsalt.utils.ctxrZsalt.utils.debugrrrrrrrrr�r��ImportErrorr�r�Zsalt.utils.win_functionsr�	getLoggerr
rBr"rrr&rOrUror�r�r�r�r�rr$r3rQr
rr?rDrEr@rNrrrr�<module>s ,���

	

79
o<zzq,x)L
@ Telegram