HM-SpringCloud微服务系列7.1【数据聚合】
- 聚合（aggregations）：实现对文档数据的统计、分析、运算。例如：
- 什么品牌的手机最受欢迎？
- 这些手机的平均价格、最高价格、最低价格？
- 这些手机每月的销售情况如何？
- 实现这些统计功能的比数据库的sql要方便的多，而且查询速度非常快，可以实现近实时搜索效果。
1 聚合的种类
- 聚合常见的有三类：
- 桶（Bucket）聚合：对文档数据做分组，并统计每组数量
- TermAggregation：按照文档字段值分组，例如按照品牌值分组、按照国家分组
- Date Histogram：按照日期阶梯分组，例如一周为一组，或者一月为一组
- 度量（Metric）聚合：对文档数据进行计算，比如：最大值、最小值、平均值等
- Avg：求平均值
- Max：求最大值
- Min：求最小值
- Stats：同时求max、min、avg、sum等
- 管道（pipeline）聚合：其它聚合的结果为基础再做聚合
- 桶（Bucket）聚合：对文档数据做分组，并统计每组数量
- 注意：参加聚合的字段必须是keyword、日期、数值、布尔类型
2 DSL实现聚合
2.1 Bucket聚合语法
-
案例需求：统计所有数据中的酒店品牌有几种，其实就是按照品牌对数据分组。此时可以根据酒店品牌的名称做聚合，也就是Bucket聚合，类型为term。
-
DSL语法
GET /hotel/_search { "size": 0, // 设置size为0，结果中不包含文档，只包含聚合结果 "aggs": { // 定义聚合 "brandAgg": { //给聚合起个名字 "terms": { // 聚合的类型，按照品牌值聚合，所以选择term "field": "brand", // 参与聚合的字段 "size": 20 // 希望获取的聚合结果数量 } } } } -
实操
点击查看代码
# 聚合功能 GET /hotel/_search { "size": 0, "aggs": { "brangAgg": { "terms": { "field": "brand", "size": 10 } } } }点击查看代码
{ "took" : 584, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 201, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "brangAgg" : { "doc_count_error_upper_bound" : 0, "sum_other_doc_count" : 39, "buckets" : [ { "key" : "7天酒店", "doc_count" : 30 }, { "key" : "如家", "doc_count" : 30 }, { "key" : "皇冠假日", "doc_count" : 17 }, { "key" : "速8", "doc_count" : 15 }, { "key" : "万怡", "doc_count" : 13 }, { "key" : "华美达", "doc_count" : 13 }, { "key" : "和颐", "doc_count" : 12 }, { "key" : "万豪", "doc_count" : 11 }, { "key" : "喜来登", "doc_count" : 11 }, { "key" : "希尔顿", "doc_count" : 10 } ] } } }
2.2 聚合结果排序
- 默认情况下，Bucket聚合会统计Bucket内的文档数量，记为_count，并且按照_count降序排序。
- 可以指定order属性，自定义聚合的排序方式：
GET /hotel/_search { "size": 0, "aggs": { "brandAgg": { "terms": { "field": "brand", "order": { "_count": "asc" // 按照_count升序排列 }, "size": 20 } } } } - 实操
2.3 限定聚合范围
-
默认情况下，Bucket聚合是对索引库的所有文档做聚合，但真实场景下，用户会输入搜索条件，因此聚合必须是对搜索结果聚合。那么聚合必须添加限定条件。
-
我们可以限定要聚合的文档范围，只要添加query条件即可：
GET /hotel/_search { "query": { "range": { "price": { "lte": 200 // 只对200元以下的文档聚合 } } }, "size": 0, "aggs": { "brandAgg": { "terms": { "field": "brand", "size": 20 } } } } -
实操
点击查看代码
# 聚合功能，限定聚合范围 GET /hotel/_search { "query": { "range": { "price": { "lte": 200 } } }, "size": 0, "aggs": { "brangAgg": { "terms": { "field": "brand", "size": 10 } } } }点击查看代码
{ "took" : 92, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 17, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "brangAgg" : { "doc_count_error_upper_bound" : 0, "sum_other_doc_count" : 0, "buckets" : [ { "key" : "如家", "doc_count" : 13 }, { "key" : "速8", "doc_count" : 2 }, { "key" : "7天酒店", "doc_count" : 1 }, { "key" : "汉庭", "doc_count" : 1 } ] } } } -
小结
2.4 Metric聚合语法
-
此前我们对酒店按照品牌分组，形成了一个个桶。
-
现在我们需要对桶内的酒店做运算，获取每个品牌的用户评分的min、max、avg等值。
-
这就要用到Metric聚合了，例如stat聚合：就可以获取min、max、avg等结果。
-
DSL语法如下：
GET /hotel/_search { "size": 0, "aggs": { "brandAgg": { "terms": { "field": "brand", "size": 20 }, "aggs": { // 是brands聚合的子聚合，也就是分组后对每组分别计算 "score_stats": { // 聚合名称 "stats": { // 聚合类型，这里stats可以计算min、max、avg等 "field": "score" // 聚合字段，这里是score } } } } } } -
这次的score_stats聚合是在brandAgg的聚合内部嵌套的子聚合。因为我们需要在每个桶分别计算。
点击查看代码
# 嵌套聚合metric GET /hotel/_search { "size": 0, "aggs": { "brangAgg": { "terms": { "field": "brand", "size": 10 }, "aggs": { "scoreAgg": { "stats": { "field": "score" } } } } } }点击查看代码
{ "took" : 443, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 201, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "brangAgg" : { "doc_count_error_upper_bound" : 0, "sum_other_doc_count" : 39, "buckets" : [ { "key" : "7天酒店", "doc_count" : 30, "scoreAgg" : { "count" : 30, "min" : 35.0, "max" : 43.0, "avg" : 37.86666666666667, "sum" : 1136.0 } }, { "key" : "如家", "doc_count" : 30, "scoreAgg" : { "count" : 30, "min" : 43.0, "max" : 47.0, "avg" : 44.833333333333336, "sum" : 1345.0 } }, { "key" : "皇冠假日", "doc_count" : 17, "scoreAgg" : { "count" : 17, "min" : 44.0, "max" : 48.0, "avg" : 46.0, "sum" : 782.0 } }, { "key" : "速8", "doc_count" : 15, "scoreAgg" : { "count" : 15, "min" : 35.0, "max" : 47.0, "avg" : 38.733333333333334, "sum" : 581.0 } }, { "key" : "万怡", "doc_count" : 13, "scoreAgg" : { "count" : 13, "min" : 44.0, "max" : 48.0, "avg" : 45.69230769230769, "sum" : 594.0 } }, { "key" : "华美达", "doc_count" : 13, "scoreAgg" : { "count" : 13, "min" : 40.0, "max" : 47.0, "avg" : 44.0, "sum" : 572.0 } }, { "key" : "和颐", "doc_count" : 12, "scoreAgg" : { "count" : 12, "min" : 44.0, "max" : 47.0, "avg" : 46.083333333333336, "sum" : 553.0 } }, { "key" : "万豪", "doc_count" : 11, "scoreAgg" : { "count" : 11, "min" : 43.0, "max" : 47.0, "avg" : 45.81818181818182, "sum" : 504.0 } }, { "key" : "喜来登", "doc_count" : 11, "scoreAgg" : { "count" : 11, "min" : 44.0, "max" : 48.0, "avg" : 46.0, "sum" : 506.0 } }, { "key" : "希尔顿", "doc_count" : 10, "scoreAgg" : { "count" : 10, "min" : 37.0, "max" : 48.0, "avg" : 45.4, "sum" : 454.0 } } ] } } } -
另外，我们还可以给聚合结果做个排序，例如按照每个桶的酒店平均分做排序：
点击查看代码
# 嵌套聚合metric GET /hotel/_search { "size": 0, "aggs": { "brangAgg": { "terms": { "field": "brand", "size": 10, "order": { "scoreAgg.avg": "desc" } }, "aggs": { "scoreAgg": { "stats": { "field": "score" } } } } } }点击查看代码
{ "took" : 926, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : { "value" : 201, "relation" : "eq" }, "max_score" : null, "hits" : [ ] }, "aggregations" : { "brangAgg" : { "doc_count_error_upper_bound" : 0, "sum_other_doc_count" : 111, "buckets" : [ { "key" : "万丽", "doc_count" : 2, "scoreAgg" : { "count" : 2, "min" : 46.0, "max" : 47.0, "avg" : 46.5, "sum" : 93.0 } }, { "key" : "凯悦", "doc_count" : 8, "scoreAgg" : { "count" : 8, "min" : 45.0, "max" : 47.0, "avg" : 46.25, "sum" : 370.0 } }, { "key" : "和颐", "doc_count" : 12, "scoreAgg" : { "count" : 12, "min" : 44.0, "max" : 47.0, "avg" : 46.083333333333336, "sum" : 553.0 } }, { "key" : "丽笙", "doc_count" : 2, "scoreAgg" : { "count" : 2, "min" : 46.0, "max" : 46.0, "avg" : 46.0, "sum" : 92.0 } }, { "key" : "喜来登", "doc_count" : 11, "scoreAgg" : { "count" : 11, "min" : 44.0, "max" : 48.0, "avg" : 46.0, "sum" : 506.0 } }, { "key" : "皇冠假日", "doc_count" : 17, "scoreAgg" : { "count" : 17, "min" : 44.0, "max" : 48.0, "avg" : 46.0, "sum" : 782.0 } }, { "key" : "万豪", "doc_count" : 11, "scoreAgg" : { "count" : 11, "min" : 43.0, "max" : 47.0, "avg" : 45.81818181818182, "sum" : 504.0 } }, { "key" : "万怡", "doc_count" : 13, "scoreAgg" : { "count" : 13, "min" : 44.0, "max" : 48.0, "avg" : 45.69230769230769, "sum" : 594.0 } }, { "key" : "君悦", "doc_count" : 4, "scoreAgg" : { "count" : 4, "min" : 44.0, "max" : 47.0, "avg" : 45.5, "sum" : 182.0 } }, { "key" : "希尔顿", "doc_count" : 10, "scoreAgg" : { "count" : 10, "min" : 37.0, "max" : 48.0, "avg" : 45.4, "sum" : 454.0 } } ] } } }
3 RestAPI实现聚合
3.1 API语法
- 聚合条件与query条件同级别，因此需要使用request.source()来指定聚合条件。
- 以品牌聚合为例，聚合条件的语法：
- 聚合的结果也与查询结果不同，API也比较特殊。不过同样是JSON逐层解析：
- 实操
@Test void testAggregation() throws IOException { // 1. 准备request SearchRequest request = new SearchRequest("hotel"); // 2. 准备DSL // 2.1 设置size，去掉文档部分（因为只想要聚合结果） request.source().size(0); // 2.2 聚合 request.source().aggregation(AggregationBuilders .terms("brandAgg") //聚合三要素：类型、名称 .field("brand") //聚合三要素：字段 .size(10) ); // 3. 发送请求，获取响应 SearchResponse response = client.search(request, RequestOptions.DEFAULT); System.out.println(response); // 4. 解析响应结果 Aggregations aggregations = response.getAggregations(); // 4.1 根据聚合名称获取聚合结果 Terms brandTerms = aggregations.get("brandAgg"); // 4.2 获取buckets List<? extends Terms.Bucket> buckets = brandTerms.getBuckets(); // 4.3 遍历buckets获取每个桶 for (Terms.Bucket bucket: buckets) { // 4.4 获取key String key = bucket.getKeyAsString(); System.out.println(key); } }