o Fh&@sddlmZddlZddlmZddlmZmZmZm Z ddlm Z ddl m Z ddlmZGdddZd d Zhd Z  dd dZ  dddZdddZdS))SequenceN) _pandas_api)CodecTable concat_tablesschema)_feather) FeatherErrorc@s6eZdZdZd ddZd ddZdd Zdd d ZdS)FeatherDataseta Encapsulates details of reading a list of Feather files. Parameters ---------- path_or_paths : List[str] A list of file names validate_schema : bool, default True Check that individual file schemas are all the same / compatible TcCs||_||_dSN)pathsvalidate_schema)self path_or_pathsr rP/var/www/html/Persson_Maskin/env/lib/python3.10/site-packages/pyarrow/feather.py__init__*s zFeatherDataset.__init__NcCsjt|jd|d}|g|_|j|_|jddD]}t||d}|jr)||||j|qt|jS)a, Read multiple feather files as a single pyarrow.Table Parameters ---------- columns : List[str] Names of columns to read from the file Returns ------- pyarrow.Table Content of the file as a table (of columns) rcolumnsN) read_tabler _tablesrr validate_schemasappendr)rr_filpathtablerrrr.s   zFeatherDataset.read_tablecCs0|j|jstd|d|jd|jdS)Nz Schema in z was different. z vs )requals ValueError)rpiecerrrrrGs zFeatherDataset.validate_schemascCs|j|dj|dS)a Read multiple Parquet files as a single pandas DataFrame Parameters ---------- columns : List[str] Names of columns to read from the file use_threads : bool, default True Use multiple threads when converting to pandas Returns ------- pandas.DataFrame Content of the file as a pandas DataFrame (of columns) r) use_threadsr to_pandas)rrr rrr read_pandasLs zFeatherDataset.read_pandas)Tr )NT)__name__ __module__ __qualname____doc__rrrr#rrrrr s  r cCsL|jdkrdS|jttfvrtd|dtd|d|jd)NrzColumn 'zg' exceeds 2GB maximum capacity of a Feather binary column. This restriction may be lifted in the futurez ' of type zU was chunked on conversion to Arrow and cannot be currently written to Feather format) num_chunkstypeextbinarystringr)namecolrrrcheck_chunked_overflow`s r/>lz4zstd uncompressedc Csvtjrtjrt|tjjr|}t|rH|dkrd}n |dkr$d}ntdt j ||d}|dkrGt |j j D] \}} ||} t| | q9n|}|dkrot|jtt|jkr^td|durftd|durntd n|dur{td r{d }n|dur|tvrtd |d tztj||||||dWdStyt|trzt|WtjyYww)a Write a pandas.DataFrame to Feather format. Parameters ---------- df : pandas.DataFrame or pyarrow.Table Data to write out as Feather format. dest : str Local destination path. compression : string, default None Can be one of {"zstd", "lz4", "uncompressed"}. The default of None uses LZ4 for V2 files if it is available, otherwise uncompressed. compression_level : int, default None Use a compression level particular to the chosen compressor. If None use the default compression level chunksize : int, default None For V2 files, the internal maximum size of Arrow RecordBatch chunks when writing the Arrow IPC file format. None means use the default, which is currently 64K version : int, default 2 Feather file version. Version 2 is the current. Version 1 is the more limited legacy format rFr3Nz%Version value should either be 1 or 2)preserve_indexz'cannot serialize duplicate column namesz2Feather V1 files do not support compression optionz0Feather V1 files do not support chunksize option lz4_framer0z compression="z " not supported, must be one of ) compressioncompression_level chunksizeversion)r have_pandas has_sparse isinstancepdSparseDataFrameto_dense is_data_framerr from_pandas enumeraternamesr/len column_namessetr is_available_FEATHER_SUPPORTED_CODECSr write_feather Exceptionstrosremoveerror) dfdestr6r7r8r9r4rir-r.rrrrIssb        rITFcKs t||||djdd|i|S)a Read a pandas.DataFrame from Feather format. To read as pyarrow.Table use feather.read_table. Parameters ---------- source : str file path, or file-like object You can use MemoryMappedFile as source, for explicitly use memory map. columns : sequence, optional Only read a specific set of columns. If not provided, all columns are read. use_threads : bool, default True Whether to parallelize reading using multiple threads. If false the restriction is used in the conversion to Pandas as well as in the reading from Feather format. memory_map : boolean, default False Use memory mapping when opening file on disk, when source is a str. **kwargs Additional keyword arguments passed on to `pyarrow.Table.to_pandas`. Returns ------- df : pandas.DataFrame The contents of the Feather file as a pandas.DataFrame )r memory_mapr r Nrr!)sourcerr rRkwargsrrr read_feathersrUcCstj|||d}|dur|St|tstdt|jdd|D}t t dd|r5| |}n t t dd|rD| |}nd d|D}td |d ||j d kr\|Stt||krf|S||S) a Read a pyarrow.Table from Feather format Parameters ---------- source : str file path, or file-like object You can use MemoryMappedFile as source, for explicitly use memory map. columns : sequence, optional Only read a specific set of columns. If not provided, all columns are read. memory_map : boolean, default False Use memory mapping when opening file on disk, when source is a str use_threads : bool, default True Whether to parallelize reading using multiple threads. Returns ------- table : pyarrow.Table The contents of the Feather file as a pyarrow.Table )use_memory_mapr Nz&Columns must be a sequence but, got {}cSsg|]}t|qSr)r)).0columnrrr szread_table..cS|tkSr )inttrrrzread_table..cSrZr )rKr\rrrr^r_cSsg|]}|jqSr)r$)rWr]rrrrY sz.Columns must be indices or names. Got columns z of types )r FeatherReaderreadr<r TypeErrorformatr)r$allmap read_indices read_namesr9sortedrFselect)rSrrRr reader column_typesrcolumn_type_namesrrrrs2      r)NNNr3)NTF)NFT)collections.abcrrLpyarrow.pandas_compatr pyarrow.librrrrlibr*pyarrowrpyarrow._featherr r r/rHrIrUrrrrrs"     B S