o Fh%@sNddlZddlZddlmZmZmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZddlmZGdddejZdZeddZGdd d ejZGd d d ejZGd d d ejZddZ ddZ!ddddZ"dede"_#dddddZ$dddddZ%dede%_#d(ddddd Z&ddd!d"d#Z'd$d%d&d'Z(dS))N)IpcReadOptionsIpcWriteOptions ReadStats WriteStatsMessage MessageReaderRecordBatchReader_ReadPandasMixinMetadataVersion Alignment read_messageread_record_batch read_schema read_tensor write_tensorget_record_batch_sizeget_tensor_sizec@s eZdZdZdddddZdS)RecordBatchStreamReadera Reader for the Arrow streaming binary format. Parameters ---------- source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object Either an in-memory buffer, or a readable file object. If you want to use memory map use MemoryMappedFile as source. options : pyarrow.ipc.IpcReadOptions Options for IPC deserialization. If None, default values will be used. memory_pool : MemoryPool, default None If None, default memory pool is used. Noptions memory_poolcCst|}|j|||ddS)Nr _ensure_default_ipc_read_options_open)selfsourcerrrL/var/www/html/Persson_Maskin/env/lib/python3.10/site-packages/pyarrow/ipc.py__init__2z RecordBatchStreamReader.__init____name__ __module__ __qualname____doc__rrrrrr"sraOParameters ---------- sink : str, pyarrow.NativeFile, or file-like Python object Either a file path, or a writable file object. schema : pyarrow.Schema The Arrow schema for data to be written to the file. options : pyarrow.ipc.IpcWriteOptions Options for IPC serialization. If None, default values will be used: the legacy format will not be used unless overridden by setting the environment variable ARROW_PRE_0_15_IPC_FORMAT=1, and the V5 metadata version will be used unless overridden by setting the environment variable ARROW_PRE_1_0_METADATA_VERSION=1. zmetadata : dict | pyarrow.KeyValueMetadata, optional Key/value pairs (both must be bytes-like) that will be stored in the file footer and are retrievable via pyarrow.ipc.open_file(...).metadata.c@s$eZdZdeZddddZdS)RecordBatchStreamWriterz.Writer for the Arrow streaming binary format NrcCst|}|j|||ddSNr'_get_legacy_format_defaultr)rsinkschemarrrrrXrz RecordBatchStreamWriter.__init__)r!r"r#_ipc_writer_class_docr$rrrrrr&Ss r&c@s"eZdZdZddddddZdS)RecordBatchFileReadera Class for reading Arrow record batch data from the Arrow binary file format Parameters ---------- source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object Either an in-memory buffer, or a readable file object. If you want to use memory map use MemoryMappedFile as source. footer_offset : int, default None If the file is embedded in some larger file, this is the byte offset to the very end of the file data options : pyarrow.ipc.IpcReadOptions Options for IPC serialization. If None, default values will be used. memory_pool : MemoryPool, default None If None, default memory pool is used. NrcCt|}|j||||ddS)N footer_offsetrrr)rrr1rrrrrrps zRecordBatchFileReader.__init__Nr rrrrr.]s r.c@s&eZdZdeZdddddZdS)RecordBatchFileWriterz/Writer to create the Arrow binary file format NrmetadatacCr/Nr4r))rr+r,rr5rrrr}szRecordBatchFileWriter.__init__)r!r"r#_ipc_file_writer_class_docr$rrrrrr3ws r3cCsh|rt|tstdt||Stj}tttj dd}tttj ddr.tj }t||dS)Nzexpected IpcWriteOptions, got ARROW_PRE_0_15_IPC_FORMAT0ARROW_PRE_1_0_METADATA_VERSION)use_legacy_formatmetadata_version) isinstancer TypeErrortyper V5boolintosenvirongetV4)rr<r;rrrr*s r*cCs*|rt|tstdt||ptS)Nzexpected IpcReadOptions, got )r=rr>r?r'rrrrs rr'cCt|||dSr()r&)r+r,rrrr new_streamsrHz5Create an Arrow columnar IPC stream writer instance zS Returns ------- writer : RecordBatchStreamWriter A writer for the given sink rcCrG)a Create reader for Arrow streaming format. Parameters ---------- source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object Either an in-memory buffer, or a readable file object. options : pyarrow.ipc.IpcReadOptions Options for IPC serialization. If None, default values will be used. memory_pool : MemoryPool, default None If None, default memory pool is used. Returns ------- reader : RecordBatchStreamReader A reader for the given source r)r)rrrrrr open_streamsrIr4cCt||||dSr6)r3)r+r,rr5rrrnew_filesrKz3Create an Arrow columnar IPC file writer instance zQ Returns ------- writer : RecordBatchFileWriter A writer for the given sink cCrJ)a Create reader for Arrow file format. Parameters ---------- source : bytes/buffer-like, pyarrow.NativeFile, or file-like Python object Either an in-memory buffer, or a readable file object. footer_offset : int, default None If the file is embedded in some larger file, this is the byte offset to the very end of the file data. options : pyarrow.ipc.IpcReadOptions Options for IPC serialization. If None, default values will be used. memory_pool : MemoryPool, default None If None, default memory pool is used. Returns ------- reader : RecordBatchFileReader A reader for the given source r0)r.)rr1rrrrr open_filesrLnthreadspreserve_indexcCs`tjj|||d}t}t||j}||Wd|S1s'wY|S)a Serialize a pandas DataFrame into a buffer protocol compatible object. Parameters ---------- df : pandas.DataFrame nthreads : int, default None Number of threads to use for conversion to Arrow, default all CPUs. preserve_index : bool, default None The default of None will store the index as a column, except for RangeIndex which is stored as metadata only. If True, always preserve the pandas index data as a column. If False, no index information is saved and the result will have a default RangeIndex. Returns ------- buf : buffer An object compatible with the buffer protocol. rMN)pa RecordBatch from_pandasBufferOutputStreamr&r, write_batchgetvalue)dfrNrObatchr+writerrrrserialize_pandass   rYT use_threadscCsHt|}t| }|}Wdn1swY|j|dS)aDeserialize a buffer protocol compatible object into a pandas DataFrame. Parameters ---------- buf : buffer An object compatible with the buffer protocol. use_threads : bool, default True Whether to parallelize the conversion using multiple threads. Returns ------- df : pandas.DataFrame The buffer deserialized as pandas DataFrame NrZ)rP BufferReaderrread_all to_pandas)bufr[ buffer_readerreadertablerrrdeserialize_pandass    rcr2))rCpyarrowrP pyarrow.librrrrrrrr r r r r rrrrrlib_RecordBatchStreamReaderrr-r7_RecordBatchStreamWriterr&_RecordBatchFileReaderr._RecordBatchFileWriterr3r*rrHr$rIrKrLrYrcrrrrs:L