How to remove _id from aggregate $match & $group query

Black Sulfur
Black Sulfur used Ask the Experts™
on
I am trying to display unique records grouped by the particular slug passed in.

My output in postman looks like this though:

	"subcats": [
	  {
		"_id": {
		  "subcategory": {
			"_id": "5d2b42c47b454712f4db7c37",
			"name": "shirts"
		  }
		}
	  }
	]

Open in new window


My desired output would be more like:

"subcats": [
  {
        "_id": "5d2b42c47b454712f4db7c37",
        "name": "shirts"
  }
]

Open in new window

An example of a product in the database:

 
       "_id": "5d39eff7a48e6e30ace831dc",
        "name": "A colourful shirt",
        "description": "A nice colourful t-shirt",
        "category": {
            "_id": "5d35faa67b19e32ab3dc91ec",
            "name": "clothing",
            "catSlug": "clothing"
        },
        "subcategory": {
            "_id": "5d2b42c47b454712f4db7c37",
            "name": "shirts",
            "catSlug": "shirts"
        },
        "price": 19
    }

Open in new window


I don't want that top level `_id` there with everything nested inside of it.

I tried using `$project` but then I just end up with an empty array.

      const products = await Product.find({ "category.catSlug": catslug }).select({
        name: 1,
        description: 1,
        price: 1,
        category: 1
      });

    const subcats = await Product.aggregate([
      { $match: { "category.catSlug": catslug } },
      { $group: { _id: { subcategory: "$subcategory" } } }
      { $project: { _id: 0, name: 1 } }
    ]);

    Promise.all([products, subcats]);
      res.status(200).json({
        products,
        subcats
      });

Open in new window

Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Most Valuable Expert 2017
Distinguished Expert 2018

Commented:
One thing before we go on - you are using Promise.all and async await - that is not a great idea.

await is going to wait for the promise to resolve on each call so you are turning what should be a parallel operation (handled by Promise.All) into a synchronous one.

Other than that I am trying to reconcile your first listing with the others.

What is it you are wanting to do - it is not clear.

Author

Commented:
Hi Julian, thanks for your response. I am glad you picked up the first issue because I want to do it right. Could you please tell me the best way to do it?

What I am trying to do is query the database and get a list of products in one query and then a list of subcategories as well which I have split into 2 queries. The results are based on the category name in the slug ie: clothing.

The list of subcategories is going to end up being a filter where a user clicks on a subcategory name and it filters products shown by that subcategory.
Most Valuable Expert 2017
Distinguished Expert 2018

Commented:
Just remove the await from the queries - and pass the promises to Promise.All.

As for the query - I am still not clear - your first snippet showed this
"subcats": [
	  {
		"_id": {
		  "subcategory": {
			"_id": "5d2b42c47b454712f4db7c37",
			"name": "shirts"
		  }
		}
	  }
	]

Open in new window

Which does not seem to match up with the values that are being returned by your query?
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Author

Commented:
If I remove await then I probably have to remove async as well and end up with this:

exports.getCatSlug = (req, res) => {
  const catslug = req.params.catslug;

  const products = Product.find({ "category.catSlug": catslug }).select({
    name: 1,
    description: 1,
    price: 1,
    category: 1
  });

  const subcats = Product.aggregate([
    { $match: { "category.catSlug": catslug } },
    { $group: { _id: { subcategory: "$subcategory.name" } } }
  ]);

  Promise.all([products, subcats]);
  res.status(200).json({
    products,
    subcats
  });

Open in new window


But then I get an error
TypeError Converting circular structure to JSON
    --> starting at object with constructor 'Server'
    |     property 's' -> object with constructor 'Object'
    |     property 'coreTopology' -> object with constructor 'Server'
    |     ...
    |     property 's' -> object with constructor

My problem with the returned query is the unnecessary "_id". I don't want it in there. I highlighted it below.

"subcats": [
        {
            "_id": {
              "subcategory": {
                  "_id": "5d2b42c47b454712f4db7c37",
                  "name": "shirts"
              }
            }
        }
      ]

Author

Commented:
So, this is where I am now. I kept async/await and just didn't use 'Promise.all'. Is that okay?

exports.getCatSlug = async (req, res) => {
  const catslug = req.params.catslug;

  const products = await Product.find({ "category.catSlug": catslug }).select({
    name: 1,
    description: 1,
    price: 1,
    category: 1
  });

  const subcats = await Product.aggregate([
    { $match: { "category.catSlug": catslug } },
    {
      $group: {
        _id: "$subcategory.name"
      }
    }
  ]);
  res.status(200).json({
    products,
    subcats
  });

Open in new window


I am happy with the products array I get back but the subcats array now looks like this:

    "subcats": [
        {
            "_id": "shirts"
        },
        {
            "_id": "shoes"
        },
        {
            "_id": "ties"
        }
    ]

Open in new window


This is better because for now I just wanted the names which I am getting, but $group is giving me _id but I would rather it said "name". Is the only way around this to create a new array?
Most Valuable Expert 2017
Distinguished Expert 2018
Commented:
There are some issues with your code that you need to be aware of.

Promise.all([products, subcats]);

Open in new window

This is not correct - the Promise.all() will return immediately and not after the promises resolve. You need to tap into the .then() of the Promise.all() for it to be useful. The reason your code works is because of the await statements - all the Promise.all() is doing is spinning the CPU's wheels.
Here is an example of how to use promises in a situation similar to yours using Promise.all();

const promise = [];

promise.push(fn1());
promise.push(fn2());
Promise.all(promise)
   .then(resp => [].concat.apply([], resp))
   .then(result => console.log(result));

function fn1() {
  return Promise.resolve('bing');
}

function fn2() {
  return Promise.resolve({name: 'Fred', surname: 'Bob'});
}

Open in new window

When you take away the await on your code the Promise.all() is not utilised and you end up trying to return a structure containing promises and not what those promises return.

The await solution will work but it is not optimal and not considered good practice. Unless the one query is dependent on the other you can reap benefits from running async functions at the same time - and then using promises to co-ordinate the completion of them. In this case probably does not make a difference but in general you use async / await in place of the following
return makeAsynchCall()
        .then(result => DoSomethingWithResult(result));

Open in new window

that would translate to
var result = await makeAsynchCall();
return DoSomethingWithResult(result);

Open in new window

In your case there is no depedency so you could simply do
const promise = [];
promise.push(Product.find( ...));
promise.push(Product.aggregate(...));
Promise.all(promise)
 .then(r => res.status(200).json([].concat.apply([], r)))

Open in new window


As for the subcats issue - what you need to do is process the data in your .then() on the Promise.all() or on the returned subcats if you are using await.
You need to iterate over subcats return and return a new structure. You can use map for this.

subcats.map(item => ({_id: item.subcategory._id, name: item.subcategory.name}));

Open in new window

And use the return from this in your res.send()
Something like this [NB: Untested]
exports.getCatSlug = (req, res) => {
  const catslug = req.params.catslug;
 
  // Get products Promise
  const products = Product.find({ "category.catSlug": catslug }).select({
    name: 1,
    description: 1,
    price: 1,
    category: 1
  });

  // Get subcats Promise
  const subcats = Product.aggregate([
    { $match: { "category.catSlug": catslug } },
    { $group: { _id: { subcategory: "$subcategory.name" } } }
  ]);

  // Wait for both products and subcats to resolve
  // Then return a promise that resolves when both products
  // subcats resolve. Set the products to whatever comes back
  // in the 0 index of the result array
  // for the subcats return, iterate over the array and return
  // a new array made up of the fields from the sub-properties
  // of each subcat item
  return Promise.all([products, subcats])
    .then(results =>  {
      res.send(200).json({
        products: results[0],
        subcats: result[1].map(item => ({_id: item.subcategory._id, name: item.subcategory.name}))
      })
    })
 }

Open in new window

Author

Commented:
Thanks Julian, I have been using async await so much that I forgot that I need to use then and catch blocks  when not using async await! Besides the map part, your code gives me the same result as to what I had changed it to (not sure if you saw that post of mine but perhaps one of your comments was related to that code? So, I just wanted to find out if you were saying this was also bad practice or just the wrong way I had done the Promise.all which totally makes sense now.


exports.getCatSlug = async (req, res) => {
  const catslug = req.params.catslug;

  const products = await Product.find({ "category.catSlug": catslug }).select({
    name: 1,
    description: 1,
    price: 1,
    category: 1
  });

  const subcats = await Product.aggregate([
    { $match: { "category.catSlug": catslug } },
    {
      $group: {
        _id: "$subcategory.name"
      }
    }
  ]);
  res.status(200).json({
    products,
    subcats
  });

Open in new window

Most Valuable Expert 2017
Distinguished Expert 2018

Commented:
The map part is what reduced the subcategory return from
"subcats": [
        {
            "_id": {
              "subcategory": {
                  "_id": "5d2b42c47b454712f4db7c37",
                  "name": "shirts"
              }
            }
        }
      ] 

Open in new window

To
"subcats": [
  {
        "_id": "5d2b42c47b454712f4db7c37",
        "name": "shirts"
  }
]

Open in new window

Besides the map part, your code gives me the same result as to what I had changed it to (not sure if you saw that post of mine but perhaps one of your comments was related to that code?
Your post was still using async / await. Mine used a Promise.all().then() approach with a map to return an altered subcats array.
Most Valuable Expert 2017
Distinguished Expert 2018

Commented:
I see you changed your query structure - which I assume is now returning the correct results.
I am not familiar with MongoDB query syntax so was not sure if your changed query was returning the data in the form that you wanted.

In event that it was not - the map code was to demonstrate how to modify an array to a different structure.

Author

Commented:
Your post was still using async / await. Mine used a Promise.all().then() approach with a map to return an altered subcats array.

Just wanted to get some clarity on the modified code below and asking if the async/await version below is still incorrect or if it is okay as I couldn't quite make out from your initial answer if it was still wrong since I no longer used Promise.all as you said:

The await solution will work but it is not optimal and not considered good practice

exports.getCatSlug = async (req, res) => {
  const catslug = req.params.catslug;

  const products = await Product.find({ "category.catSlug": catslug }).select({
    name: 1,
    description: 1,
    price: 1,
    category: 1
  });

  const subcats = await Product.aggregate([
    { $match: { "category.catSlug": catslug } },
    {
      $group: {
        _id: "$subcategory.name"
      }
    }
  ]);
  res.status(200).json({
    products,
    subcats
  });

Open in new window


If the above isn't good practice or not a good way of doing it then I will use Promise.all going forward as per your code but was just wanted to make sure I understood what you were saying properly.
Most Valuable Expert 2017
Distinguished Expert 2018

Commented:
Yes, with the await as you have it you have turned your async calls into synch calls - so you don't need Promise.all as the promises will resolve before continuing.

Author

Commented:
Great, thanks for all your help, it is much appreciated!
Most Valuable Expert 2017
Distinguished Expert 2018

Commented:
You are welcome.

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial