商品索引维护架构文档
概述
- 商品属于高频搜索数据，直接查询数据库压力过大，因此我们采用ElasticSearch（以下简称ES）来创建映射和索引，并进行搜索
- ES会将商品名称进行分词，用户在用关键字搜索商品时会按照商品名称拆分出来的内容进行匹配
- Javashop系统目前支持的ES版本为6.4到7.10之间，推荐使用7.9.3版本
创建索引流程
说明
对于商品索引的创建，系统提供了两种方式：手动和自动
手动创建：平台管理员可以登录管理端，进入运营 -> 维护 -> 商品索引页面中，手动创建索引（此方式针对系统内所有的商品，相当于商品索引初始化）
自动创建：商品数据变更（新增、修改、上下架等）时，系统会自动对商品所以进行更新维护
商品索引的维护是通过异步消息来进行处理的
当商品数据变更时，系统会通过RabbitMQ发送一条消息，然后消费者类（Consumer）接收到消息后，通过操作类型（新增、修改、上下架等）来判断如何处理商品索引
流程图
索引初始化
自动更新索引
索引内容
com.enation.app.javashop.model.goodssearch.GoodsIndex
public class GoodsIndex {
/**
* 商品ID
*/
@Id
private Long goodsId;
/**
* 商品名称
*/
@Field(type = FieldType.Text, analyzer = EsSettings.IK_MAX_WORD)
private String name;
/**
* 商品缩略图
*/
@Field(type = FieldType.Text)
private String thumbnail;
/**
* 商品小图
*/
@Field(type = FieldType.Text)
private String small;
/**
* 销售数量
*/
@Field(type = FieldType.Integer)
private Integer buyCount;
/**
* 商家ID
*/
@Field(type = FieldType.Long)
private Long sellerId;
/**
* 商家名称
*/
@Field(type = FieldType.Text)
private String sellerName;
/**
* 店铺分组ID
*/
@Field(type = FieldType.Long)
private Long shopCatId;
/**
* 店铺分组ID路径
*/
@Field(type = FieldType.Text)
private String shopCatPath;
/**
* 评论数量
*/
@Field(type = FieldType.Integer)
private Integer commentNum;
/**
* 商品评分
*/
@Field(type = FieldType.Double)
private Double grade;
/**
* 优惠价格（预留字段，暂时未用到）
*/
@Field(type = FieldType.Double)
private double discountPrice;
/**
* 商品价格
*/
@Field(type = FieldType.Double)
private double price;
/**
* 品牌ID
*/
@Field(type = FieldType.Long)
private Long brand;
/**
* 分类ID
*/
@Field(type = FieldType.Long)
private Long categoryId;
/**
* 分类ID路径
*/
@Field(type = FieldType.Text)
private String categoryPath;
/**
* 是否被删除 0：删除，1：未删除
*/
@Field(type = FieldType.Integer)
private Integer disabled;
/**
* 上下架状态 0：下架，1：上架
*/
@Field(type = FieldType.Integer)
private Integer marketEnable;
/**
* 审核状态 0：待审核，1：无需审核或审核已通过，2：审核未通过
*/
@Field(type = FieldType.Integer)
private Integer isAuth;
/**
* 商品描述
*/
@Field(type = FieldType.Text)
private String intro;
/**
* 商品搜索优先级
*/
@Field(type = FieldType.Integer)
private Integer priority;
/**
* 兑换商品需要的积分
*/
@Field(type = FieldType.Integer)
private Integer exchangePoint;
/**
* 是否开启积分兑换 0：否，1：是
*/
@Field(type = FieldType.Integer)
private Integer pointDisable;
/**
* 商品类型 NORMAL：普通商品，VIRTUAL：虚拟商品
*/
@Field(type = FieldType.Text)
private String goodsType;
/**
* 是否为自营店铺商品 0：否，1：是
*/
@Field(type = FieldType.Integer)
private Integer selfOperated;
/**
* 是否为推荐商品 0：否，1：是
*/
@Field(type = FieldType.Integer)
private Integer recommend;
/**
* 商品参数信息
*/
@Field(type = FieldType.Nested, index = true, store = true)
private List<Param> params;
//get&set略
}
重点字段说明：
商品名称（name）：ES会对商品名称进行分词，因此需要指定分词粒度（analyzer = EsSettings.IK_MAX_WORD），具体可参考索引分词架构
商品分类路径（categoryPath）与店铺分组路径（shopCatPath）：这两个数据在生成索引时，会将原数据进行Hex加密，然后再存入索引
加密前 加密后 0|1|3|10| 307c317c337c31307c 参数信息（params）：FieldType.Nested属于嵌套类型，可以让array类型的Object独立索引和查询
核心类图展示
代码展示
发送异步消息
商品索引的维护都是通过消费者类（Consumer）来进行异步处理的，发送消息代码如下：
初始化索引消息发送
IndexCreateMessage indexCreateMessage = new IndexCreateMessage();
messageSender.send(indexCreateMessage);
商品信息变更消息发送（以新增商品为例）
GoodsChangeMsg goodsChangeMsg = new GoodsChangeMsg(new Long[]{goods.getGoodsId()},GoodsChangeMsg.ADD_OPERATION);
messageSender.send(goodsChangeMsg);
GoodsChangeMsg中包含的操作类型有：
/**
* 添加
*/
public final static int ADD_OPERATION = 1;
/**
* 修改
*/
public final static int UPDATE_OPERATION = 2;
/**
* 删除
*/
public final static int DEL_OPERATION = 3;
/**
* 下架
*/
public final static int UNDER_OPERATION = 4;
/**
* 还原
*/
public final static int REVERT_OPERATION = 5;
/**
* 放入回收站
*/
public final static int INRECYCLE_OPERATION = 6;
/**
* 商品成功审核
*/
public final static int GOODS_VERIFY_SUCCESS = 7;
/**
* 商品失败审核
*/
public final static int GOODS_VERIFY_FAIL = 8;
/**
* 商品优先级变更
*/
public final static int GOODS_PRIORITY_CHANGE = 9;
消息接收类
初始化索引消息接收类
com.enation.app.javashop.consumer.core.receiver.GoodsIndexInitReceiver
@Component
public class GoodsIndexInitReceiver {
@Autowired
private GoodsIndexInitDispatcher goodsIndexInitDispatcher;
/**
* 处理初始化商品索引消息
*
* @param indexCreateMessage
*/
@RabbitListener(bindings = @QueueBinding(
value = @Queue(value = AmqpExchange.INDEX_CREATE + "_QUEUE"),
exchange = @Exchange(value = AmqpExchange.INDEX_CREATE, type = ExchangeTypes.FANOUT)
))
public void initGoodsIndex(IndexCreateMessage indexCreateMessage) {
goodsIndexInitDispatcher.dispatch(indexCreateMessage);
}
}
索引变更消息接收类
com.enation.app.javashop.consumer.core.receiver.GoodsChangeReceiver
@Component
public class GoodsChangeReceiver {
@Autowired
GoodsChangeDispatcher goodsChangeDispatcher;
/**
* 处理商品信息变化消息
*
* @param goodsChangeMsg
*/
@RabbitListener(bindings = @QueueBinding(
value = @Queue(value = AmqpExchange.GOODS_CHANGE + "_QUEUE"),
exchange = @Exchange(value = AmqpExchange.GOODS_CHANGE, type = ExchangeTypes.FANOUT)
))
public void goodsChange(GoodsChangeMsg goodsChangeMsg) {
goodsChangeDispatcher.dispatch(goodsChangeMsg);
}
}
处理索引消费者
初始化索引
com.enation.app.javashop.message.consumer.goodssearch.GoodsIndexInitConsumer
@Override
public void createGoodsIndex() {
debugger.log("开始生成索引");
String key = TaskProgressConstant.GOODS_INDEX;
try {
/** 获取商品数 */
int goodsCount = this.goodsClient.queryGoodsCount();
/** 生成任务进度 */
progressManager.taskBegin(key, goodsCount);
//删除所有索引
goodsIndexClient.deleteAll();
Long start= System.currentTimeMillis();
//生成商品索引
LongStream.iterate(1L, i -> i + 1)
.parallel()
.limit(this.getPages())
.forEach(goodsIndexClient::addAll);
//任务结束
progressManager.taskEnd(key, "索引生成完成");
logger.debug("索引生成时间"+ (System.currentTimeMillis()-start));
logger.debug("索引生成完成");
} catch (Exception e) {
debugger.log("索引生成异常");
progressManager.taskError(key, "生成索引异常，请联系运维人员");
this.logger.error("生成索引异常：", e);
}
}
索引变更
com.enation.app.javashop.message.consumer.goodssearch.GoodsChangeIndexConsumer
@Override
public void goodsChange(GoodsChangeMsg goodsChangeMsg) throws IOException {
//获取商品ID信息
Long[] goodsIds = goodsChangeMsg.getGoodsIds();
//获取商品变动操作类型
int operationType = goodsChangeMsg.getOperationType();
List<Map<String, Object>> list = goodsClient.getGoodsAndParams(goodsIds);
//添加
if (GoodsChangeMsg.ADD_OPERATION == operationType) {
if (list != null && list.size() > 0) {
goodsIndexClient.addIndex(list.get(0));
}
} else if (GoodsChangeMsg.UPDATE_OPERATION == operationType
|| GoodsChangeMsg.INRECYCLE_OPERATION == operationType
|| GoodsChangeMsg.REVERT_OPERATION == operationType
|| GoodsChangeMsg.GOODS_VERIFY_SUCCESS == operationType
|| GoodsChangeMsg.GOODS_VERIFY_FAIL == operationType
|| GoodsChangeMsg.UNDER_OPERATION == operationType ) {//修改(修改，还原，放入购物车，审核)
if (list != null) {
for (Map<String, Object> map : list) {
goodsIndexClient.updateIndex(map);
}
}
}
else if (GoodsChangeMsg.DEL_OPERATION == operationType ) {
//删除
if (list != null) {
for (Map<String, Object> map : list) {
goodsIndexClient.deleteIndex(map);
}
}
}
}
商品索引增删改
com.enation.app.javashop.service.goodssearch.impl.GoodsIndexManagerImpl
public class GoodsIndexManagerImpl {
/**
* 新增商品索引
*
* @param goods 商品数据
*/
@Override
@Transactional(value = "goodsTransactionManager", propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
public boolean addIndex(Map goods) {
//获取商品名称
String goodsName = goods.get("goods_name").toString();
//配置文件中定义的索引名字
String indexName = elasticJestConfig.getIndexName() + "_" + EsSettings.GOODS_INDEX_NAME;
List<String> wordsList = toWordsList(goodsName);
// 分词入库
this.wordsToDb(wordsList);
GoodsIndex goodsIndex = this.getSource(goods);
boolean result = ElasticOperationUtil.insert(jestClient, indexName, goodsIndex);
if (!result) {
logger.error("为商品[" + goodsName + "]生成索引异常");
debugger.log("为商品[" + goodsName + "]生成索引异常");
}
return result;
}
/**
* 更新商品索引
*
* @param goods 商品数据
*/
@Override
@Transactional(value = "goodsTransactionManager", propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
public void updateIndex(Map goods) {
//删除
this.deleteIndex(goods);
//添加
this.addIndex(goods);
}
/**
* 删除商品索引
*
* @param goods 商品数据
*/
@Override
@Transactional(value = "goodsTransactionManager", propagation = Propagation.REQUIRED, rollbackFor = Exception.class)
public void deleteIndex(Map goods) {
///配置文件中定义的索引名字
String indexName = elasticJestConfig.getIndexName() + "_" + EsSettings.GOODS_INDEX_NAME;
//构建删除请求
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(QueryBuilders.termQuery("goodsId", goods.get("goods_id").toString()));
boolean result = ElasticOperationUtil.deleteByQuery(jestClient, indexName, searchSourceBuilder.toString());
if (!result) {
throw new ServiceException(GoodsErrorCode.E310.code(), "删除商品索引异常");
}
String goodsName = goods.get("goods_name").toString();
List<String> wordsList = toWordsList(goodsName);
this.deleteWords(wordsList);
}
/**
* 封装成内存需要格式数据
*
* @param goods
* @return
*/
protected GoodsIndex getSource(Map goods) {
GoodsIndex goodsIndex = new GoodsIndex();
goodsIndex.setGoodsId(StringUtil.toLong(goods.get("goods_id").toString(), 0));
goodsIndex.setName(goods.get("goods_name").toString());
goodsIndex.setThumbnail(goods.get("thumbnail") == null ? "" : goods.get("thumbnail").toString());
goodsIndex.setSmall(goods.get("small") == null ? "" : goods.get("small").toString());
Double p = goods.get("price") == null ? 0d : StringUtil.toDouble(goods.get("price").toString(), 0d);
goodsIndex.setPrice(p);
Double discountPrice = goods.get("discount_price") == null ? 0d : StringUtil.toDouble(goods.get("discount_price").toString(), 0d);
goodsIndex.setDiscountPrice(discountPrice);
goodsIndex.setBuyCount(goods.get("buy_count") == null ? 0 : StringUtil.toInt(goods.get("buy_count").toString(), 0));
goodsIndex.setSellerId(StringUtil.toLong(goods.get("seller_id").toString(), 0));
//店铺分组
goodsIndex.setShopCatId(goods.get("shop_cat_id") == null ? 0 : StringUtil.toLong(goods.get("shop_cat_id").toString(), 0));
goodsIndex.setShopCatPath(goods.get("shop_cat_path") == null ? "" : goods.get("shop_cat_path").toString());
if (goodsIndex.getShopCatId() != 0) {
ShopCatDO shopCat = shopCatClient.getModel(goodsIndex.getShopCatId());
if (shopCat != null) {
goodsIndex.setShopCatPath(HexUtils.encode(shopCat.getCatPath()));
}
}
goodsIndex.setSellerName(goods.get("seller_name").toString());
goodsIndex.setCommentNum(goods.get("comment_num") == null ? 0 : StringUtil.toInt(goods.get("comment_num").toString(), 0));
goodsIndex.setGrade(goods.get("grade") == null ? 100 : StringUtil.toDouble(goods.get("grade").toString(), 100d));
goodsIndex.setBrand(goods.get("brand_id") == null ? 0 : StringUtil.toLong(goods.get("brand_id").toString(), 0));
goodsIndex.setCategoryId(goods.get("category_id") == null ? 0 : StringUtil.toLong(goods.get("category_id").toString(), 0));
CategoryDO cat = categoryManager.getModel(Long.valueOf(goods.get("category_id").toString()));
goodsIndex.setCategoryPath(HexUtils.encode(cat.getCategoryPath()));
goodsIndex.setDisabled(StringUtil.toInt(goods.get("disabled").toString(), 0));
goodsIndex.setMarketEnable(StringUtil.toInt(goods.get("market_enable").toString(), 0));
goodsIndex.setIsAuth(StringUtil.toInt(goods.get("is_auth").toString(), 0));
goodsIndex.setIntro(goods.get("intro") == null ? "" : goods.get("intro").toString());
goodsIndex.setSelfOperated(goods.get("self_operated") == null ? 0 : StringUtil.toInt(goods.get("self_operated").toString(), 0));
if (i18n) {
//添加国际化索引
goodsIndex = this.fillI18n(goodsIndex, goods);
}//endif
//添加商品优先级维度
goodsIndex.setPriority(goods.get("priority") == null ? 1 : StringUtil.toInt(goods.get("priority").toString(), 1));
//商品类型
goodsIndex.setGoodsType(goods.get("goods_type") == null ? GoodsType.NORMAL.name() : goods.get("goods_type").toString());
//商品积分兑换
goodsIndex.setPointDisable(goods.get("point_disable") == null ? 0 : StringUtil.toInt(goods.get("point_disable").toString(), 0));
goodsIndex.setExchangePoint(goods.get("exchange_point") == null ? 0 : StringUtil.toInt(goods.get("exchange_point").toString(), 0));
//是否推荐商品
goodsIndex.setRecommend(goods.get("recommend") == null ? 0 : StringUtil.toInt(goods.get("recommend").toString(), 0));
//参数维度,已填写参数
List<Map> params = (List<Map>) goods.get("params");
List<Param> paramsList = this.convertParam(params);
goodsIndex.setParams(paramsList);
return goodsIndex;
}
}