Mongo DB with 25M Records


I have a requirement to load 25M records in a single collection and I did that. When I access through my REST / Phalcon API Call, the server getting shutdown and it through the following error message from the cursor.

"Failed to connect to: localhost:27017: Remote server has closed the connection 500 Internal Server Error - MongoConnectionException"

Can you please suggest me, how to do I access large sets ? How do I split the database if incase.

Thanks, Raja K

25M is a large set, if you try to read all of the rows intoa json response, your server will run out of memory. Create a pagination logic and only serve a few hundred records per request.

DB partitioning is also useful, but I dont know if mongo supports it

Hello there...

Not familiar with MongoDB but I found a very useful and accurate article called SSIS MongoDB Integration Where all points are described clearly in deep manner.

edited 7d ago

That's really not going to work. 25 million of something seems like a batch processing you want to have, and not live API response.

Client -> API -> MongoDB on that pipeline all three must be able to process that many data.

If you really need to do that - use some form of buffering. So pipeline will process only chunks of data and dispatch it directly to the client (persistent connection).