小菜鸟

索引模板

elasticsearch索引模板是创建好一个索引参数设置settings和映射mapping的模板，在创建索引的时候指定模板名称就可以使用模板定义好的参数设置和映射

创建索引模板

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

PUT _template/template_log  // template_log是表示模板名称

{
    "index_patterns": [
        "*"
    ], // 对哪些索引生效，我这是通用模板，对所有索引生效
    "order": 0,
    "settings": {
        "number_of_replicas": 0,
        "translog": {
            "sync_interval": "60s",
            "durability": "async"
        },
        "refresh_interval": "300s"
    }
}

修改模板不会影响已创建的索引，只会对新创建的索引生效

elasticsearch连接查询

在elasticsearch中提供了两种关联关系，一种是嵌套关系，一种是父子关系，可以进行这两种关联的查询

对象类型存在的问题

在说明连接查询之前，先展示一下对象类型进行查询时的问题

有一条文档

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

{
    "name": "技术大会",
    "events": [
      {
        "date": "2022-10-10",
        "title": "java架构大会"
      },
      {
        "date": "2022-08-08",
        "title": "hadoop大会"
      },
      {
        "date": "2022-02-02",
        "title": "elasticsearch大会"
      }
    ]
}

假设查询hadoop是否在2022-10-01和2022-12-31范围内

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

{
    "query": {
        "bool": {
            "must": [
                {
                    "term": {
                        "events.title": "hadoop"
                    }
                },
                {
                    "range": {
                        "events.date": {
                            "from": "2022-10-01",
                            "to": "2022-12-31"
                        }
                    }
                }
            ]
        }
    }
}

按理说不应该查到数据，但是

elasticsearch系统状态查询

elasticsearch提供了一套cat APIs来展示系统的状态

如GET /_cat/allocation?v

• aliases 别名信息

  1 2 3 4 5

  alias index filter routing.index routing.search alias1 test1 - - - alias2 test1 * - - alias3 test1 - 1 1 alias4 test1 - 2 1,2

• allocation 数据节点的分片快照以及使用的磁盘空间大小

  1 2	shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 5 260b 47.3gb 43.4gb 100.7gb 46 127.0.0.1 127.0.0.1 CSUXak2

• count 节点的文档总数

  1 2	epoch timestamp count 1475868259 15:24:19 121

• fielddata 每个数据节点上正在使用的堆内存

  1 2 3

  id host ip node field size Nqk-6inXQq-OxUfOUI8jNQ 127.0.0.1 127.0.0.1 Nqk-6in body 544b Nqk-6inXQq-OxUfOUI8jNQ 127.0.0.1 127.0.0.1 Nqk-6in soul 480b

• health 健康情况

  1 2	epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1475871424 16:17:04 elasticsearch green 1 1 5 5 0 0 0 0 - 100.0%

• indices 节点下的索引信息，可以得到索引的分片、文档、存储大小等信息

  1 2 3

  health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open twitter u8FNjxh8Rfy_awN11oDKYQ 1 1 1200 0 88.1kb 88.1kb green open twitter2 nYFWZEO7TUiOjLQXBaYJpA 5 0 0 0 260b 260b

• master 显示master节点的概要信息

  1 2	id host ip node YzWoH_2BT-6UjVGDyPdqYg 127.0.0.1 127.0.0.1 YzWoH_2

• nodeattrs 显示自定义节点属性

  1 2 3 4

  node host ip attr value ... node-0 127.0.0.1 127.0.0.1 testattr test ...

• nodes 节点相关信息

  1 2	ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 127.0.0.1 65 99 42 3.07 mdi * mJw06l1

• pending tasks 集群任务接口

  1 2 3 4 5 6 7 8

  insertOrder timeInQueue priority source 1685 855ms HIGH update-mapping [foo][t] 1686 843ms HIGH update-mapping [foo][t] 1693 753ms HIGH refresh-mapping [foo][[t]] 1688 816ms HIGH update-mapping [foo][t] 1689 802ms HIGH update-mapping [foo][t] 1690 787ms HIGH update-mapping [foo][t] 1691 773ms HIGH update-mapping [foo][t]

• plugins 集群中插件安装的信息

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

  name component version description U7321H6 analysis-icu 6.8.23 The ICU Analysis plugin integrates the Lucene ICU module into Elasticsearch, adding ICU-related analysis components. U7321H6 analysis-kuromoji 6.8.23 The Japanese (kuromoji) Analysis plugin integrates Lucene kuromoji analysis module into elasticsearch. U7321H6 analysis-nori 6.8.23 The Korean (nori) Analysis plugin integrates Lucene nori analysis module into elasticsearch. U7321H6 analysis-phonetic 6.8.23 The Phonetic Analysis plugin integrates phonetic token filter analysis with elasticsearch. U7321H6 analysis-smartcn 6.8.23 Smart Chinese Analysis plugin integrates Lucene Smart Chinese analysis module into elasticsearch. U7321H6 analysis-stempel 6.8.23 The Stempel (Polish) Analysis plugin integrates Lucene stempel (polish) analysis module into elasticsearch. U7321H6 analysis-ukrainian 6.8.23 The Ukrainian Analysis plugin integrates the Lucene UkrainianMorfologikAnalyzer into elasticsearch. U7321H6 discovery-azure-classic 6.8.23 The Azure Classic Discovery plugin allows to use Azure Classic API for the unicast discovery mechanism U7321H6 discovery-ec2 6.8.23 The EC2 discovery plugin allows to use AWS API for the unicast discovery mechanism. U7321H6 discovery-file 6.8.23 Discovery file plugin enables unicast discovery from hosts stored in a file. U7321H6 discovery-gce 6.8.23 The Google Compute Engine (GCE) Discovery plugin allows to use GCE API for the unicast discovery mechanism. U7321H6 ingest-attachment 6.8.23 Ingest processor that uses Apache Tika to extract contents U7321H6 mapper-annotated-text 6.8.23 The Mapper Annotated_text plugin adds support for text fields with markup used to inject annotation tokens into the index. U7321H6 mapper-murmur3 6.8.23 The Mapper Murmur3 plugin allows to compute hashes of a field's values at index-time and to store them in the index. U7321H6 mapper-size 6.8.23 The Mapper Size plugin allows document to record their uncompressed size at index time. U7321H6 store-smb 6.8.23 The Store SMB plugin adds support for SMB stores.

• recovery 正在进行和以前完成的索引分片的回收率

  1 2

  index shard time type stage source_host source_node target_host target_node repository snapshot files files_recovered files_percent files_total bytes bytes_recovered bytes_percent bytes_total translog_ops translog_ops_recovered translog_ops_percent twitter 0 13ms store done n/a n/a 127.0.0.1 node-0 n/a n/a 0 0 100% 13 0 0 100% 9928 0 0 100.0%

• repositories 集群中注册的快照库

  1 2 3

  id type repo1 fs repo2 s3

• thread pool 集群节点中的线程池统计

  1 2 3 4 5 6 7

  node-0 analyze 0 0 0 ... node-0 fetch_shard_started 0 0 0 node-0 fetch_shard_store 0 0 0 node-0 flush 0 0 0 ... node-0 write 0 0 0

• shards 索引分片的使用情况

  1	twitter 0 p STARTED 3014 31.1mb 192.168.56.10 H5dfFeA

• segments 索引分片中低水平段的信息

  1 2 3

  index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound test 3 p 127.0.0.1 _0 0 1 0 3kb 2042 false true 7.7.3 true test1 3 p 127.0.0.1 _0 0 1 0 3kb 2042 false true 7.7.3 true

• snapshots 仓库的快照信息

  1 2 3

  id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards snap1 FAILED 1445616705 18:11:45 1445616978 18:16:18 4.6m 1 4 1 5 snap2 SUCCESS 1445634298 23:04:58 1445634672 23:11:12 6.2m 2 10 0 10

• templates 模板信息

  1 2 3 4

  name index_patterns order version template0 [te*] 0 template1 [tea*] 1 template2 [teak*] 2 7

elasticsearch复杂查询

有时候查询语句会非常复杂，这时候肯定只能使用DSL查询了，使用多个简单子句合并成一个复杂的查询语句

查询DSL的格式为

• 查询：以query参数开始，用于检查内容与条件是否匹配
• 过滤：以filter参数开始，不计算匹配得分，只是简单的决定文档是否匹配，主要用于过滤结构化数据
• 叶查询子句：在特定的字段上查找特定的值，如match，term 和range
• 复合查询子句：包含其他叶查询或复合查询子句，以合理的方式结合多条查询，如bool或dis_max查询，或者改变查询行为，如constant_score查询

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

{
  "bool":{ // 合并多个条件的布尔逻辑
    "must":{ // 多个查询条件完全匹配，相当于and
      "match":{
        "field":""
      }
    },
    "must_not":{ // 多个查询条件都不匹配，相当于not
      "match":{
        "field":""
      }
    },
    "should":{ // 有一个条件匹配，相当于or
      "match":{
        "field":""
      }
    }
  }
}

elasticsearch索引

新建索引

1

PUT blog

返回结果acknowledged的值为true表示新建索引成功

映射

映射是用于进行字段类型确认的，将每个字段匹配为一种确定的数据类型

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

// 查看文档结果
GET video/_mapping/default


{
    "video": {
        "mappings": {
            "default": {
                "properties": {
                    "age": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    },
                    "doc_id": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    },
                    "doc_title": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    },
                    "duration": {
                        "type": "long"
                    },
                    "keywords": {
                        "type": "text",
                        "fields": {
                            "keyword": {
                                "type": "keyword",
                                "ignore_above": 256
                            }
                        }
                    },
                    "number": {
                        "type": "long"
                    }
                }
            }
        }
    }
}