Toggle navigation
Toggle navigation
此项目
正在载入...
Sign in
xuning
/
sherpaonnx
转到一个项目
Toggle navigation
项目
群组
代码片段
帮助
Toggle navigation pinning
Project
Activity
Repository
Pipelines
Graphs
Issues
0
Merge Requests
0
Wiki
Network
Create a new issue
Builds
Commits
Authored by
Fangjun Kuang
2023-10-13 10:51:59 +0800
Browse Files
Options
Browse Files
Download
Email Patches
Plain Diff
Committed by
GitHub
2023-10-13 10:51:59 +0800
Commit
323f532ad2225ef37102f41c83e24f8067a468c3
323f532a
1 parent
efd3cd33
Fix symbol table for byte bpe (#361)
隐藏空白字符变更
内嵌
并排对比
正在显示
1 个修改的文件
包含
10 行增加
和
1 行删除
sherpa-onnx/csrc/symbol-table.cc
sherpa-onnx/csrc/symbol-table.cc
查看文件 @
323f532
...
...
@@ -60,7 +60,16 @@ void SymbolTable::Init(std::istream &is) {
}
assert
(
!
sym
.
empty
());
assert
(
sym2id_
.
count
(
sym
)
==
0
);
// for byte bpe, after replacing ▁ with a space, whose ascii is also 0x20,
// there is a conflict between the real byte 0x20 and ▁, so we disable
// the following check.
//
// Note: Only id2sym_ matters as we use it to convert ID to symbols.
if
(
sym
!=
" "
)
{
assert
(
sym2id_
.
count
(
sym
)
==
0
);
}
assert
(
id2sym_
.
count
(
id
)
==
0
);
sym2id_
.
insert
({
sym
,
id
});
...
...
请
注册
或
登录
后发表评论