File: //opt/saltstack/salt/lib/python3.10/site-packages/gitdb/__pycache__/stream.cpython-310.pyc
o
;j�k � @ s. d dl mZ d dlZd dlZd dlZd dlZd dlmZmZm Z m
Z
m Z d dl m
Z
mZmZmZmZ d dlmZmZ d dlmZ dZz
d dlmZ d ZW n eyU Y nw d
Z G d d � d e�Z G d
d� de�Z G dd� d�Z G dd� de �Z G dd� de �Z!G dd� de �Z"G dd� d�Z#G dd� d�Z$dS )� )�BytesION)�msb_size� stream_copy�apply_delta_data�connect_deltas� delta_types)�allocate_memory� LazyMixin�make_sha�write�close)� NULL_BYTE�
BYTE_SPACE)� force_bytesF)� apply_deltaT) �DecompressMemMapReader�FDCompressedSha1Writer�DeltaApplyReader�
Sha1Writer�FlexibleSha1Writer�ZippedStoreShaWriterr �FDStream�
NullStreamc @ s~ e Zd ZdZdZdZd dd�Zdd� Zd d
� Zd d � Z e
d dd��Z dd� Z dd� Z
dd� Zeedd�fdd�Zd dd �ZdS ) r a� Reads data in chunks from a memory map and decompresses it. The client sees
only the uncompressed data, respective file-like read calls are handling on-demand
buffered decompression accordingly
A constraint on the total size of bytes is activated, simulating
a logical file within a possibly larger physical memory area
To read efficiently, you clearly don't want to read individual bytes, instead,
read a few kilobytes at least.
**Note:** The chunk-size should be carefully selected as it will involve quite a bit
of string copying due to the way the zlib is implemented. Its very wasteful,
hence we try to find a good tradeoff between allocation time and number of
times we actually allocate. An own zlib implementation would be good here
to better support streamed reading - it would only need to keep the mmap
and decompress it into chunks, that's all ... ) �_m�_zip�_buf�_buflen�_br�_cws�_cwe�_s�_close�_cbr�_phii Nc C sR || _ t�� | _d| _d| _|dur|| _d| _d| _d| _ d| _
d| _ || _ dS )z|Initialize with mmap for stream reading
:param m: must be content data - use new if you have object data and no sizeNr F)
r �zlib�
decompressobjr r r r r r r r" r# r! )�self�m�close_on_deletion�size� r* �@/opt/saltstack/salt/lib/python3.10/site-packages/gitdb/stream.py�__init__E s
z DecompressMemMapReader.__init__c C s |dksJ �| � � d S )Nr )�_parse_header_info�r& �attrr* r* r+ � _set_cache_U s z"DecompressMemMapReader._set_cache_c C s | � � d S �N)r �r& r* r* r+ �__del__[ � z DecompressMemMapReader.__del__c C s~ d}|| _ | �|�}|�t�}|d|� �t�\}}t|�}|| _ d| _|d7 }t||d� �| _ t
|�| | _ d| _ ||fS )z�If this stream contains object data, parse the header info and skip the
stream to a point where each read will yield object content
:return: parsed type_string, sizei Nr � T)
r �read�findr
�splitr �intr r r �lenr r# )r& Zmaxb�hdrZhdrend�typr) r* r* r+ r- ^ s
z)DecompressMemMapReader._parse_header_infoFc C s" t ||d�}|�� \}}|||fS )a� Create a new DecompressMemMapReader instance for acting as a read-only stream
This method parses the object header from m and returns the parsed
type and size, as well as the created stream instance.
:param m: memory map on which to operate. It must be object data ( header + contents )
:param close_on_deletion: if True, the memory map will be closed once we are
being deletedr )r r- )r& r' r( �instr< r) r* r* r+ �new{ s
zDecompressMemMapReader.newc C � | j S )z8:return: random access compatible data we are working on)r r2 r* r* r+ �data� s zDecompressMemMapReader.datac C s* | j rt| jd�r| j�� d| _ dS dS )z�Close our underlying stream of compressed bytes if this was allowed during initialization
:return: True if we closed the underlying stream
:note: can be called safely
r FN)r! �hasattrr r r2 r* r* r+ r � s
�z DecompressMemMapReader.closec C s� | j | jkrJ| jjsJd| _ t| jd�r(| jjtjkr'| �t j
� | jjtjksn | jjsF| j t | j
�krF| �t j
� | jjsF| j t | j
�ks4| j| _ | j S )z�
:return: number of compressed bytes read. This includes the bytes it
took to decompress the header ( if there was one )r �status)r r r � unused_datarA rB r$ ZZ_OKr6 �mmap�PAGESIZEr" r: r r2 r* r* r+ �compressed_bytes_read� s �� �z,DecompressMemMapReader.compressed_bytes_read�SEEK_SETr c C sZ |dks |t tdd�krtd��t�� | _d | _ | _ | _| _ | j
r+d| _
| ` dS dS )zgAllows to reset the stream to restart reading
:raise ValueError: If offset and whence are not 0r rG �Can only seek to position 0FN) �getattr�os�
ValueErrorr$ r% r r r r r" r# r �r& �offset�whencer* r* r+ �seek� s
�zDecompressMemMapReader.seek���c C s� |dk r | j | j }n t|| j | j �}|dkrdS d}| jrR| j|kr:| j�|�}| j|8 _| j|7 _|S | j�� }|| j8 }| j| j7 _d| _d | _| jj}|rg| jt |� | _
| j
| | _n | j
}| j| _
|| | _| j| j
dk r�| j
d | _| j | j
| j� }| j
t |� | _| j� ||�}t
tdtj�dv r�tjdks�t | jj�}n t | jj�t | jj� }| jt |�| 7 _| jt |�7 _|r�|| }|r�t |�t |� |k r�| j| j k r�|| �|t |� �7 }|S )Nr5 r � � �ZLIB_RUNTIME_VERSION)z1.2.7z1.2.5�darwin)r r �minr r r6 r �unconsumed_tailr r: r r �
decompressrI r$ � ZLIB_VERSION�sys�platformrC r" )r& r) Zdat�tailZcwsZindataZdcompdatZunused_datalenr* r* r+ r6 � sN
$ zDecompressMemMapReader.readr1 �F)rP )�__name__�
__module__� __qualname__�__doc__� __slots__Z
max_read_sizer, r0 r3 r- � classmethodr> r@ r rF rI rJ rO r6 r* r* r* r+ r . s
1r c @ s� e Zd ZdZdZdZdd� Zdd� Zdd � Ze seZ
neZ
dd d �Z e e
d
d
�fdd�Zedd� �Zedd� �Zedd� �Zedd� �ZdS )r a� A reader which dynamically applies pack deltas to a base object, keeping the
memory demands to a minimum.
The size of the final object is only obtainable once all deltas have been
applied, unless it is retrieved from a pack index.
The uncompressed Delta has the following layout (MSB being a most significant
bit encoded dynamic size):
* MSB Source Size - the size of the base against which the delta was created
* MSB Target Size - the size of the resulting data after the delta was applied
* A list of one byte commands (cmd) which are followed by a specific protocol:
* cmd & 0x80 - copy delta_data[offset:offset+size]
* Followed by an encoded offset into the delta data
* Followed by an encoded size of the chunk to copy
* cmd & 0x7f - insert
* insert cmd bytes from the delta buffer into the output stream
* cmd == 0 - invalid operation ( or error in delta stream )
)�_bstream� _dstreams�
_mm_target�_sizer i���c C s: t |�dks
J d��|d | _t|dd� �| _d| _dS )z�Initialize this instance with a list of streams, the first stream being
the delta to apply on top of all following deltas, the last stream being the
base object onto which to apply the deltasr5 z+Need at least one delta and one base streamrP Nr )r: rc �tuplerd r )r& � stream_listr* r* r+ r, h s
zDeltaApplyReader.__init__c C s� t | j�dkr | �|�S t| j�}|�� dkr!d| _td�| _d S |�� | _t| j�| _t| jj �}t
| jj |j | jj dt
j � | jj }|�||� | j�d� d S )Nr5 r � )r: rd �_set_cache_brute_r Zrboundrf r re rc r) r r6 r rD rE ZapplyrO )r& r/ Zdcl�bbufr r* r* r+ � _set_cache_too_slow_without_cr s
z.DeltaApplyReader._set_cache_too_slow_without_cc C sl t � }d}| jD ]&}|�d�}t|�\}}t||�\}}|�||d� |||f� t||�}q| jj} |}t| j�dkrCt| |� } }t | �}
t
| jj|
j | dt j
� t |�} d} tt|�t| j��D ]I\\}
}}}}t |j| �}|� |
� t
|j|j |jdt j
� dt� v r�t|
|| � n t|
||t|�| j � | |
}
} |
�d� | �d� |} qd|
| _| | _dS )z*If we are here, we apply the actual deltasr i Nr5 ri �
c_apply_delta)�listrd r6 r �append�maxrc r) r: r r r rD rE �zip�reversed�globalsrm r rO re rf )r& r/ Zbuffer_info_listZmax_target_sizeZdstream�bufrM Zsrc_sizeZ target_sizeZ base_sizerk ZtbufZfinal_target_size�dbufZddatar* r* r+ rj � s:
$
z"DeltaApplyReader._set_cache_brute_r c C sB | j | j }|dk s||kr|}| j�|�}| jt|�7 _|S )Nr5 )rf r re r6 r: )r& �countZblr@ r* r* r+ r6 � s zDeltaApplyReader.readrG c C s6 |dks |t tdd�krtd��d| _| j�d� dS )zhAllows to reset the stream to restart reading
:raise ValueError: If offset and whence are not 0r rG rH N)rI rJ rK r re rO rL r* r* r+ rO � s zDeltaApplyReader.seekc C s< t |�dk r
td��|d jtv rtd|d j ��| |�S )a
Convert the given list of streams into a stream which resolves deltas
when reading from it.
:param stream_list: two or more stream objects, first stream is a Delta
to the object that you want to resolve, followed by N additional delta
streams. The list's last stream must be a non-delta stream.
:return: Non-Delta OPackStream object whose stream can be used to obtain
the decompressed resolved data
:raise ValueError: if the stream list cannot be handled� zNeed at least two streamsrP zNCannot resolve deltas if there is no base object stream, last one was type: %s)r: rK �type_idr �type)�clsrh r* r* r+ r> � s
�zDeltaApplyReader.newc C � | j jS r1 )rc ry r2 r* r* r+ ry � zDeltaApplyReader.typec C r{ r1 )rc rx r2 r* r* r+ rx r| zDeltaApplyReader.type_idc C r? )z3:return: number of uncompressed bytes in the stream)rf r2 r* r* r+ r) s zDeltaApplyReader.sizeN�r )r] r^ r_ r` ra Zk_max_memory_mover, rl rj � has_perf_modr0 r6 rI rJ rO rb r> �propertyry rx r) r* r* r* r+ r B s(
"K
r c @ s. e Zd ZdZdZdd� Zdd� Zd dd �Zd
S ) r zpSimple stream writer which produces a sha whenever you like as it degests
everything it is supposed to write�sha1c C s t � | _d S r1 )r
r� r2 r* r* r+ r, 2 r4 zSha1Writer.__init__c C s | j �|� t|�S )z{:raise IOError: If not all bytes could be written
:param data: byte object
:return: length of incoming data)r� �updater: �r& r@ r* r* r+ r 7 s zSha1Writer.writeFc C s |r| j �� S | j �� S )z]:return: sha so far
:param as_hex: if True, sha will be hex-encoded, binary otherwise)r� � hexdigest�digest)r& Zas_hexr* r* r+ �shaD s
zSha1Writer.shaNr\ )r] r^ r_ r` ra r, r r� r* r* r* r+ r , s
r c @ s$ e Zd ZdZdZdd� Zdd� ZdS )r zZWriter producing a sha1 while passing on the written bytes to the given
write function�writerc C s t �| � || _d S r1 )r r, r� )r& r� r* r* r+ r, T s
zFlexibleSha1Writer.__init__c C s t �| |� | �|� d S r1 )r r r� r� r* r* r+ r X s zFlexibleSha1Writer.writeN)r] r^ r_ r` ra r, r r* r* r* r+ r N s
r c @ sP e Zd ZdZdZdd� Zdd� Zdd� Zd d
� Ze e
d d �fd
d�Z dd� Z dS )r z=Remembers everything someone writes to it and generates a sha)rt rq c C s$ t �| � t� | _t�tj�| _d S r1 )r r, r rt r$ � compressobj� Z_BEST_SPEEDrq r2 r* r* r+ r, b s
z ZippedStoreShaWriter.__init__c C s t | j|�S r1 )rI rt r. r* r* r+ � __getattr__g r4 z ZippedStoreShaWriter.__getattr__c C s$ t �| |�}| j�| j�|�� |S r1 )r r rt rq �compress)r& r@ �alenr* r* r+ r j s zZippedStoreShaWriter.writec C s | j �| j�� � d S r1 )rt r rq �flushr2 r* r* r+ r p s zZippedStoreShaWriter.closerG r c C s0 |dks |t tdd�krtd��| j�d� dS )z`Seeking currently only supports to rewind written data
Multiple writes are not supportedr rG rH N)rI rJ rK rt rO rL r* r* r+ rO s s zZippedStoreShaWriter.seekc C s
| j �� S )zA:return: string value from the current stream position to the end)rt �getvaluer2 r* r* r+ r� { s
z ZippedStoreShaWriter.getvalueN)
r] r^ r_ r` ra r, r� r r rI rJ rO r� r* r* r* r+ r ] s r c s< e Zd ZdZdZed�Z� fdd�Zdd� Zdd � Z � Z
S )
r z�Digests data written to it, making the sha available, then compress the
data and write it to the file descriptor
**Note:** operates on raw file descriptors
**Note:** for this to work, you have to use the close-method of this instance)�fdr� rq z+Failed to write all bytes to filedescriptorc s" t � �� || _t�tj�| _d S r1 )�superr, r� r$ r� r� rq �r& r� �� __class__r* r+ r, � s
z FDCompressedSha1Writer.__init__c C s> | j �|� | j�|�}t| j|�}|t|�kr| j�t|�S )zZ:raise IOError: If not all bytes could be written
:return: length of incoming data)r� r� rq r� r r� r: �exc)r& r@ ZcdataZ
bytes_writtenr* r* r+ r � s z FDCompressedSha1Writer.writec C s. | j �� }t| j|�t|�kr| j�t| j�S r1 )rq r� r r� r: r� r )r& � remainderr* r* r+ r � s
z FDCompressedSha1Writer.close) r] r^ r_ r` ra �IOErrorr� r, r r �
__classcell__r* r* r� r+ r � s r c @ sF e Zd ZdZdZdd� Zdd� Zddd �Zd
d � Zd d
� Z dd� Z
dS )r z�A simple wrapper providing the most basic functions on a file descriptor
with the fileobject interface. Cannot use os.fdopen as the resulting stream
takes ownership��_fd�_posc C s || _ d| _d S �Nr r� r� r* r* r+ r, � s
zFDStream.__init__c C s$ | j t|�7 _ t�| j|� d S r1 )r� r: rJ r r� r� r* r* r+ r � s zFDStream.writer c C s: |dkr t j�| j�}t �| j|�}| jt|�7 _|S r� )rJ �path�getsizeZ _filepathr6 r� r� r: )r& rv �bytesr* r* r+ r6 � s
z
FDStream.readc C r? r1 )r� r2 r* r* r+ �fileno� � zFDStream.filenoc C r? r1 )r� r2 r* r* r+ �tell� r� z
FDStream.tellc C s t | j� d S r1 )r r� r2 r* r* r+ r � s zFDStream.closeNr} ) r] r^ r_ r` ra r, r r6 r� r� r r* r* r* r+ r � s
r c @ s0 e Zd ZdZe� Zd
dd�Zdd� Zdd� Zd S ) r zVA stream that does nothing but providing a stream interface.
Use it like /dev/nullr c C s dS )N� r* )r& r) r* r* r+ r6 � � zNullStream.readc C s d S r1 r* r2 r* r* r+ r � r� zNullStream.closec C s t |�S r1 )r: r� r* r* r+ r � s zNullStream.writeNr} ) r] r^ r_ r` rg ra r6 r r r* r* r* r+ r � s
r )%�ior rD rJ rY r$ Z gitdb.funr r r r r Z
gitdb.utilr r r
r r Z gitdb.constr
r Zgitdb.utils.encodingr r~ Zgitdb_speedups._perfr rm � ImportError�__all__r r r r r r r r r* r* r* r+ �<module> s8 � k"#("