Much of the power and flexibility in Muut IO comes from the ability to manipulate metadata associated with posts. Metadata can be used to filter and sort content as well. This article will cover the basics of how our metadata works and some examples of what you can build using it.

Metadata Shmetadata — what it is and why you should love it

Usually, when building an application you’ll figure out what data you need for your application to run, and then design the scheme for your data. To do this you will find out what fields you will need, figure out what ways you will want to organize your data, then make sure you create appropriate indexes so that your queries don’t drag, and you should definitely benchmark how you'll access this data with a populated store to tune performance.

Whether you are using a SQL or NoSQL datastore you will need to plan out how you will access the data in both a scalable and future proof way. The data planning phase can be quite daunting, and often something as simple as changing the ways you want to access your data can require some massive refactoring down the road.

Using the schema-less metadata enables you to hit the ground running in a fraction of the time it would take you otherwise

With our metadata functionality, we abstract away this entire process and allow you to store a structured document of data along with any of your community content. This same metadata can then be used to filter and sort your content. The equivalent to “where” and “order by” statements in SQL without having to worry about indexes and being able to add fields to your data, arbitrarily.

Metadata in Muut is schema-less, so you can have different fields in different documents, add new fields as necessary and just consume your content without having to stress about the data model or how it will perform.

Let’s say you have an online store where people are discussing products. You could tag a thread with “questions”, and perhaps you have a metadata field for the product category “electronics”, and another for status. You could have a back office interface that queries for all open questions in electronics for your electronics support team, updating the metadata status to “answered” once resolved.

The ability to ad-hoc add fields, and create new functionality as you go lets you get your product live quickly without having to design a future-proof model accounting for all future use cases. Using the schema-less metadata enables you to hit the ground running in a fraction of the time it would take you otherwise.

Let’s take a look at this magical metadata

Muut metadata is strongly typed JSON data. Normally JSON data doesn’t have strong typing like compiled languages do. Since we have to do a lot of indexing magic under the hood to allow you your arbitrary querying and filtering we have to be a bit more strict on the data than JSON would typically allow.

The types available in Muut IO metadata:

  • long - Long numeric values. If you need to represent decimal values, then you will need to use the double type.
  • string - String types are treated as a list of string keys with spaces delimiting items. So this: [“two”,”keys”] is identical to “two keys”.
  • date - The date format can be set using either an ISOz formatted datetime OR the epoch timestamp in seconds. When you receive metadata back it’s always returned as an epoch timestamp regardless of the format used to set it.
  • boolean - Pretty self explanatory.
  • object - Objects allow you to structure your data to organize it better.
  • double - When the value distribution for long just doesn't cut it.
  • geo - You can store geographic coordinates in these fields. Coordinates take the form of an array of two floats (lat and lon). For example: [37.77,-122.4] would be the approximate coordinates for San Francisco.

Use metadata by prefixing the name of a property with the type followed by an underscore. You can use underscores anywhere else as well. We only care about the first one.

A property might be named date_some_date_field, and that field will display as a date. Setting anything other than a valid date will result in an error.

The types and prefixes are as follows:

  • long: long_
  • string: str_
  • date: date_
  • boolean: bool_
  • object: obj_
  • double: dbl_
  • geo: geo_

Metadata might end up looking like this:

{
  str_assigned: "johndoe",
  str_tags: ["support","questions","billing"],
  str_status: "open",
  long_priority: 3,
  geo_user_location: [37.77,-122.4],
  dbl_percent_complete: 0.5,
  date_when_assigned: 1479159658,
  obj_source: {
     str_product: "widget",
     str_category: "electronics"
  }
}

As you can see the str_tags field has an array of strings while the other string fields don’t. All types accept arrays safely. You don’t need to be consistent either. You can use an array in one thread and a single value in another. You can imagine single values as a single value array.

You can safely use arrays in objects as well so:

{
  obj_emotions: [
     { str_type: ":smiley1:", str_from: "johndoe"},
     { str_type: ":smiley2:", str_from: "janedoe"}
  ]
}

The above is perfectly legal, and you can still sort or filter by values in the nested objects. Perhaps you only want to receive results that have :smiley1:. Well, that’s no problem at all.

Adding and using metadata

When a post is made you can pass along a meta property that contains metadata for the post. All metadata exists within a scope that tells you the visibility of the data. At the moment the only scope available is public.

This means that when you create a post or reply using the REST API, io-client, or the Muut client, you simply add a property called meta with a value such as:

{
  public: { str_property: "value" }
}

If you wish to update the metadata on a post after it’s already been made you can update it using the Meta API endpoint.

Metadata is available every time the post data itself is provided. It’s sent along with REST API calls (listings and events API), thread listings via the io-client or Muut client, realtime notifications, as well as webhooks. If you retrieve the post you will also always be sent the metadata associated with that post as a meta field.

Filtering / Sorting queries

Using metadata gives you an immense amount of power, but things get exciting when you start filtering or sorting data based on metadata values. Sorting and Filtering are available for all trials and all plans of at least the M level and above.

Filtering and sorting are handled via query objects. They are ultimately rather simple. Something like this:

{
  version: 1,
  path: "/projectname/some/path",
    filter: {
    "meta.public.str_tags": ["some","tags"],
    "meta.public.str_type": "question",
    "meta.public.long_priority: { gt: 2 },
    "meta.public.geo_location: 
      {distance: "100km", from: [37.77,-122.4]}
  },
  sort: ["meta.public.long_priority","post.date desc"]
}

Let’s go through this line by line.

version: 1 - Because the syntax and structure of the query object may change as we add functionality, we require a syntax version to ensure that as functionality grows, we don’t break any of your existing queries.

path: "/projectname/some/path” - This is the context under which to run this query. The results will only contain children from this node in your hierarchical data. You can read more about Muut's hierarchical data on this blog post.

filter: { - This section is optional, but if it exists it will only return results that match.

”meta.public.str_tags": ["some","tags"] - If you list multiple items, then any items matching will mean a result. Threads that have both either “some” or “tags” value in the str_tags property will return. You cannot yet make AND queries.

"meta.public.str_type": "question” - If you are looking for a single value, you can represent it as a string or as a single valued array such as [“question”].

"meta.public.long_priority: { gt: 2 } - On long and double fields you can query for ranges using gt lt get and lte. In this case we will only return results where the long_priority property is greater than 2. You can use more than one of the parameters to search for a value that resides between two values. Keep in mind that the value must match each condition so { lt: 2, gt: 2} would always return zero results since no value can be greater than and less than 2.

"meta.public.geo_location: {distance: "100km", from: [37.77,-122.4]} - If you’ve stored any coordinates you can filter results that are within a certain distance from a point. The distance is simply a string taking the form [numeric value][unit]. You simply provide the distance and then the lat/lon coordinates in an array. In this case we’re only returning results with a geo_location for points within 100km of San Francisco. If the geo_location value in the thread is an array, then it will pass the filter if any of the point pass the filter. The units available are:

  • mi - miles
  • ft - feet
  • km - kilometers
  • m - meters

sort: ["meta.public.long_priority","post.date desc”] - In the sort field you simply list the fields in an array that you want to sort the results by. The default order if not provided is ascending. If you wish to change the sort direction you just add desc to the field. You can also list asc if you want to say the sort direction, explicitly. If no sort field is provided, then the results are sorted by post.last_reply desc by default. The available fields to sort by are any metadata fields. You also have post.date (when the thread was created) and post.last_reply (when the last reply was).

note String types are not case sensitive when filtering. The case is retained but when filtering the case of the metadata value and the case of the value in your filter query will not have any impact.

How to execute your query

Right now, there are two ways you can run your query.:

You can pass a query object to the REST API content endpoint. You pass a property named query instead of a path with the query structure from above.

The other way is using the io-client (our low-level browser Muut library). Rather than initialize the client with muutio('myproject', fn) you’d do: muutio({query: query_object}, fn) where query_object is the object we describe above. You can query for threads after initialization as well with a call like so:

api.call('threads', {
  query: {
    filter: { 'meta.public.str_tags': ['important'] },
    path: '/acme/support',
    version: 1
  }
}, function(threads) {

  // threads loaded here

})

An upcoming version of the full Muut client will also support queries so you can embed certain results and it’ll “just work.”

Up next…

We’re aggressively pushing the capabilities of our metadata system to give you an incredible amount of power without having to worry about the nuts and bolts

We’re aggressively pushing the capabilities of our metadata system to give you an incredible amount of power without having to worry about the nuts and bolts of how to handle these types of operations at scale.

You can look forward to improved query semantics for more complex boolean statements, aggregating metadata values for replies, new trend values for sorting and filtering ("most likes this week" for example).

More visibility scopes other than public will be forthcoming as well.

We love metadata

Metadata and all the power that comes with it is something we’ve been working on making real for quite some time. With the flexibility, it provides along with the hierarchical structure of Muut data you can design almost any type of community/social application.

If the functionality provided doesn’t quite get you all the way there, we’d love to hear from you. It’s our goal to make a system flexible enough to build your application while giving you the kind of performance you could only hope to achieve with a massive technical investment. You get to market faster with a product that simply flies.

— Your Muut team

Get started with Muut IO

muut.io

Next Post Previous Post