<div>
<a href="https://pynative.com/python-cursor-fetchall-fetchmany-fetchone-to-read-rows-from-table/" rel="ugc"><strong>cursor.fetchmany(size)</strong> </a>is probably what you want
</div>


ocean O

<div>
fetchmany should do what you want, isn't it working?
</div>


<div>
Thanks for reply!<br />
Actually I need do something like : if in the DB there are records of 11000, I need fetch three times, the first two 5000, the third time 1000. &nbsp;And put this 11000 row into one csv file . How can I do it? Thanks
</div>


<div>
Why not fetch all the records in one go?<br />
<br />
Are you processing them before the data in the CSV file?
</div>


<div>
<blockquote>
I need fetch three times, the first two 5000, the third time 1000. &nbsp;And put this 11000 row into one csv file . How can I do it?
</blockquote>
<br />
what's the <b>time gap</b> for these 3 processes we are talking about? we also have mentioned how frequent you wish to do this.<br />
<br />
if there's no time gap, then try fetch all records in one go as mentioned.<br />
<br />
if there's a time gap, perhaps you can try to save the output in 3 separate CSVs, and then combine them together in your final step.
</div>


<div>
Thanks for the reply.<br />
I am calling a postgresql db in python, since data in some table is very big which can affect the performance. So I am thinking to use something like &quot;cursor fetch batch&quot;, so I can fetch three times instead of one time, then I can combine all the data and return one csv file.<br />
<br />
Thanks
</div>


<div>
I don't really have a hands-on experience on postgresql + python implementation, but I guess in order to fetch large data from a very large table.<br />
<br />
perhaps you could try to:<br />
<br />
1. Replicate the data needed to be exported into another temporary table.<br />
2. Build a FIFO process model, when your ETL (or extraction python code) is running, mark the records that have been exported<br />
3. When it reaches certain amount of records being exported, combine the CSVs into a single CSV.<br />
4. Delete such marked records in temporary table when it's necessary<br />
<br />
in short, you may need more than one python scripts to handle these processes.
</div>


<div>
Hi, Norie:<br />
Thanks for the suggestion.<br />
Do you have any example that need fetch data 3 times and then put into dataframe?<br />
<br />
Thanks
</div>


<div>
I had written a function like below, but I am not sure how to put all the data into one dataframe. For example if it fetchs 3 times, how can I append all the data into one dataframe? &nbsp;Thanks<br />
<br />
def db_execute(sql, execution_type):<br />
<br />
&nbsp; &nbsp; try:<br />
&nbsp; &nbsp; &nbsp; &nbsp; conn = get_connection_handle()<br />
&nbsp; &nbsp; &nbsp; &nbsp; conn.autocommit = True<br />
&nbsp; &nbsp; &nbsp; &nbsp; cursor = conn.cursor()<br />
&nbsp; &nbsp; &nbsp; &nbsp; cursor.execute(sql)<br />
&nbsp; &nbsp; &nbsp; &nbsp; if execution_type == _READ:<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; while True:<br />
<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; results = cursor.fetchmany(4000)<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if not results:<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; break<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; for r in results:<br />
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; data = pd.DataFrame(results)<br />
<br />
&nbsp; &nbsp; &nbsp; &nbsp; cursor.close()<br />
&nbsp; &nbsp; &nbsp; &nbsp; close_connection_handle(co<wbr />nn)<br />
&nbsp; &nbsp; &nbsp; &nbsp; return data
</div>


<div>
<div class="content wysiwyg-content">
Hi, I have a question about python.<br />
I have a line of code in py file: results = cursor.fetchall()<br />
But I need change to retrieve the rows in some chunks ( ex: 5000 rows) instead of &nbsp;&quot;fetchall&quot;<br />
<br />
For example: &nbsp; &nbsp; rows = cursor.fetchmany(5000)<br />
<br />
How can I do it?<br />
Thanks
</div>
</div>


Hi, I have a question about python.
I have a line of code in py file: results = cursor.fetchall()
But I need change to retrieve the rows in some chunks ( ex: 5000 rows) instead of  "fetchall"

For example:     rows = cursor.fetchmany(5000)

How can I do it?
Thanks

Python sql

Python is a widely used general-purpose, high-level programming language. Its design philosophy emphasizes code readability, and its syntax allows programmers to express concepts in fewer lines of code than would be possible in other languages. Python supports multiple programming paradigms, including object-oriented, imperative and functional programming or procedural styles. It features a dynamic type system and automatic memory management and has a large and comprehensive set of standard libraries, including NumPy, SciPy, Django, PyQuery, and PyLibrary.

Python

PostgreSQL is an object-relational database management system with an emphasis on extensibility and standards-compliance. As a database server, its primary function is to store data securely, supporting best practices, and to allow for retrieval at the request of other software applications. PostgreSQL implements the majority of the SQL2011 standard, is ACID-compliant and transactional using multiversion concurrency control (MVCC), and handles complex SQL queries using many indexing methods that are not available in other databases. It is cross-platform and runs on many operating systems including Linux, FreeBSD, OS X, Solaris, and Microsoft Windows.

PostgreSQL

SQL (Structured Query Language) is designed to be used in conjunction with relational database products as of a means of working with sets of data. SQL consists of data definition, data manipulation, and procedural elements. Its scope includes data insert, query, update and delete, schema creation and modification, and data access control.