分类 ❤····随手 下的文章

总体思路是使用数据挖掘中的挖掘频繁项原理来挖掘毒品之间的关系、人口普查数据与毒品之间的关系。
以下是论文第二部分的主干。

Glossary & Symbols

Glossary

数据立方体 data cube
维 dimension
分层 hierarchy hierarchy is a mapping sequence that maps the underlying conceptual set to a higher level
频繁模式 frequent pattern
关联规则 association rule
支持度 support
置信度 confidence
提升度 lift
出现频度 instances
频繁项集 frequent itemset

Symbols

公式

confidence(A => B) = P(A|B) = support_count(AUB)/support_count(A)

Model 1

1 Data Analysis

The data in MCM_NFLIS_Data.xlsx can be abstracted as data cut with 4 dimension, the 4 demension are time, position, substance name and substance report number. In time dimension we have one hierarchy —— year, in position dimension we have two hierarchy —— state and county, in substance name dimension we have one hierarchy with 69 kinds of substances and each substance has its report number in drug report number dimension.

1.1 The Intuitive Concept of States

In order to have an intuitive concept of states, we improve position dimension to state level, and statistic on the total number of drug report.
drug-identification-count.png
The figure above shows the number of drugs found in five states in the past seven years. In terms of the number and proportion of drug detection, both OH and PA are on the rise, so we should focus on the data of these two states.

1.2 The Intuitive Concept of Substance

In order to have an intuitive concept of substance, we slice to substance level, and statistic on the total number of each substance in the past seven years.
substance-name-identification-counts.png
The figure above shows the top 13 substances number found in the past seven years. In terms of the number and proportion of these substance, Fentanyl, Carfentanil, Furanyl fentanyl, U-47700, Acryl fentanyl and Tramadol are on the rise, Heroin is always in a high level.

2 Data Mining

2.1 Data Preparation

We dice this data in two dimension time and position to get a list of substance name and report number in each county, then we roll-up report number dimension to get if a county has selected substance with true or false flag.

2.2 Data Mining

At first, we calculate support for each substance, if the instances of substance is higher than MIN_SUPPORT, this substance is a one item frequent itemset.
Then we use the frequent itemset at first step to generate the two item set. And calculate support for each set, if the instances of set is higher than MIN_SUPPORT, this set is a two item frequent itemset. Repeat this step until there is no support degree lager than MIN_SUPPORT.
1-itemsets-support.png
The chart above shows the distribution of support degree for one item frequent itemset. We find that there is a clear demarcation at support from 0.1 to 0.2.
2-itemsets-support.png
The chart above shows the distribution of support degree for two item frequent itemset. We find that there is a clear demarcation at support near 0.1.
3-itemsets-support.png
The chart above shows the distribution of support degree for two item frequent itemset. We find that there is a clear demarcation at support near 0.1.

It can be seen that the support degree of one item frequent itemset is bounded at 0.2-0.1, and the support degree between two item and three item frequent itemset is bounded at 0.1. In order to improve the stability and accuracy of the model, we set the MIN_SUPPORT to 0.1 so that this model can more accurately analyze and pay attention to the relatively serious situation, and can warn as early as possible.
We get 11 one item frequent itemset, 55 two item frequent itemset and 165 three item frequent itemset.
看篇幅情况,放适量的下方数据
Below are some of the frequent itemset.

==================================================
frequent 1-itemsets        support
==================================================
['Oxycodone']      0.85
['Hydrocodone']      0.81
['Heroin']      0.74
['Buprenorphine']      0.68
['Morphine']      0.57
['Methadone']      0.49
['Fentanyl']      0.40
['Tramadol']      0.38
['Hydromorphone']      0.32
['Oxymorphone']      0.32
['Codeine']      0.28
==================================================
frequent 2-itemsets        support
==================================================
['Oxycodone', 'Hydrocodone']      0.76
['Oxycodone', 'Heroin']      0.68
['Oxycodone', 'Buprenorphine']      0.65
['Hydrocodone', 'Heroin']      0.64
['Hydrocodone', 'Buprenorphine']      0.63
['Buprenorphine', 'Heroin']      0.59
['Oxycodone', 'Morphine']      0.55
['Hydrocodone', 'Morphine']      0.54
['Morphine', 'Heroin']      0.48
['Methadone', 'Oxycodone']      0.48
['Morphine', 'Buprenorphine']      0.48
['Methadone', 'Hydrocodone']      0.47
['Methadone', 'Heroin']      0.42
['Methadone', 'Buprenorphine']      0.41
['Fentanyl', 'Heroin']      0.38
['Methadone', 'Morphine']      0.38
['Oxycodone', 'Fentanyl']      0.38
['Oxycodone', 'Tramadol']      0.36
['Fentanyl', 'Hydrocodone']      0.36
['Tramadol', 'Hydrocodone']      0.35
['Fentanyl', 'Buprenorphine']      0.35
['Tramadol', 'Buprenorphine']      0.33
['Tramadol', 'Heroin']      0.33
['Oxycodone', 'Oxymorphone']      0.32
['Oxycodone', 'Hydromorphone']      0.32
['Hydrocodone', 'Hydromorphone']      0.31
['Oxymorphone', 'Hydrocodone']      0.31
['Tramadol', 'Morphine']      0.30
['Oxymorphone', 'Buprenorphine']      0.29
['Fentanyl', 'Morphine']      0.29
['Hydromorphone', 'Heroin']      0.29
['Oxymorphone', 'Heroin']      0.28
['Hydromorphone', 'Buprenorphine']      0.28
['Codeine', 'Oxycodone']      0.27
['Codeine', 'Hydrocodone']      0.27
['Hydromorphone', 'Morphine']      0.27
['Oxymorphone', 'Morphine']      0.27
['Methadone', 'Tramadol']      0.27
['Codeine', 'Heroin']      0.26
['Codeine', 'Buprenorphine']      0.25
['Methadone', 'Hydromorphone']      0.24
['Methadone', 'Fentanyl']      0.24
['Methadone', 'Oxymorphone']      0.24
['Fentanyl', 'Tramadol']      0.24
['Codeine', 'Morphine']      0.24
['Methadone', 'Codeine']      0.21
['Tramadol', 'Hydromorphone']      0.19
['Oxymorphone', 'Tramadol']      0.18
['Oxymorphone', 'Hydromorphone']      0.18
['Codeine', 'Tramadol']      0.18
['Oxymorphone', 'Fentanyl']      0.18
['Codeine', 'Fentanyl']      0.18
['Fentanyl', 'Hydromorphone']      0.17
['Codeine', 'Oxymorphone']      0.16
['Codeine', 'Hydromorphone']      0.16
==================================================
frequent 3-itemsets        support
==================================================
['Oxycodone', 'Hydrocodone', 'Heroin']      0.62
['Oxycodone', 'Hydrocodone', 'Buprenorphine']      0.61
['Oxycodone', 'Buprenorphine', 'Heroin']      0.57
['Hydrocodone', 'Buprenorphine', 'Heroin']      0.54
['Oxycodone', 'Hydrocodone', 'Morphine']      0.53
['Oxycodone', 'Morphine', 'Heroin']      0.47
['Oxycodone', 'Morphine', 'Buprenorphine']      0.47
['Methadone', 'Hydrocodone', 'Oxycodone']      0.46
['Hydrocodone', 'Morphine', 'Buprenorphine']      0.46
['Hydrocodone', 'Morphine', 'Heroin']      0.46
['Morphine', 'Buprenorphine', 'Heroin']      0.42
['Oxycodone', 'Methadone', 'Buprenorphine']      0.41
['Oxycodone', 'Methadone', 'Heroin']      0.41
['Methadone', 'Hydrocodone', 'Buprenorphine']      0.40
['Methadone', 'Hydrocodone', 'Heroin']      0.40
['Methadone', 'Oxycodone', 'Morphine']      0.38
['Methadone', 'Hydrocodone', 'Morphine']      0.37
['Methadone', 'Buprenorphine', 'Heroin']      0.37
['Oxycodone', 'Fentanyl', 'Heroin']      0.36
['Oxycodone', 'Fentanyl', 'Hydrocodone']      0.35
['Oxycodone', 'Hydrocodone', 'Tramadol']      0.35
['Fentanyl', 'Heroin', 'Hydrocodone']      0.34
['Methadone', 'Morphine', 'Buprenorphine']      0.34
['Fentanyl', 'Buprenorphine', 'Heroin']      0.34
['Oxycodone', 'Fentanyl', 'Buprenorphine']      0.34
['Methadone', 'Morphine', 'Heroin']      0.34
['Oxycodone', 'Tramadol', 'Buprenorphine']      0.33
['Fentanyl', 'Buprenorphine', 'Hydrocodone']      0.32
['Oxycodone', 'Tramadol', 'Heroin']      0.32
['Tramadol', 'Buprenorphine', 'Hydrocodone']      0.32
['Tramadol', 'Hydrocodone', 'Heroin']      0.31
['Oxycodone', 'Hydrocodone', 'Hydromorphone']      0.31
['Oxycodone', 'Oxymorphone', 'Hydrocodone']      0.31
['Tramadol', 'Buprenorphine', 'Heroin']      0.30
['Oxycodone', 'Tramadol', 'Morphine']      0.29
['Hydrocodone', 'Morphine', 'Tramadol']      0.29
['Oxycodone', 'Oxymorphone', 'Buprenorphine']      0.29
['Oxycodone', 'Hydromorphone', 'Heroin']      0.28
['Oxymorphone', 'Buprenorphine', 'Hydrocodone']      0.28
['Oxycodone', 'Fentanyl', 'Morphine']      0.28
['Oxycodone', 'Oxymorphone', 'Heroin']      0.28
['Fentanyl', 'Morphine', 'Hydrocodone']      0.28
['Hydrocodone', 'Hydromorphone', 'Heroin']      0.28
['Oxycodone', 'Hydromorphone', 'Buprenorphine']      0.27
['Oxymorphone', 'Hydrocodone', 'Heroin']      0.27
['Fentanyl', 'Morphine', 'Heroin']      0.27
['Tramadol', 'Morphine', 'Buprenorphine']      0.27
['Hydrocodone', 'Hydromorphone', 'Buprenorphine']      0.27
['Oxycodone', 'Hydromorphone', 'Morphine']      0.27
['Codeine', 'Hydrocodone', 'Oxycodone']      0.27
['Hydrocodone', 'Hydromorphone', 'Morphine']      0.27
['Fentanyl', 'Buprenorphine', 'Morphine']      0.27
['Methadone', 'Tramadol', 'Oxycodone']      0.27
['Oxycodone', 'Oxymorphone', 'Morphine']      0.27
['Tramadol', 'Morphine', 'Heroin']      0.26
['Oxymorphone', 'Morphine', 'Hydrocodone']      0.26
['Methadone', 'Hydrocodone', 'Tramadol']      0.26
['Oxymorphone', 'Buprenorphine', 'Heroin']      0.26
['Codeine', 'Oxycodone', 'Heroin']      0.26
['Hydromorphone', 'Buprenorphine', 'Heroin']      0.25
['Codeine', 'Hydrocodone', 'Heroin']      0.25
['Codeine', 'Oxycodone', 'Buprenorphine']      0.25
['Methadone', 'Tramadol', 'Buprenorphine']      0.25
['Oxymorphone', 'Buprenorphine', 'Morphine']      0.25
['Codeine', 'Hydrocodone', 'Buprenorphine']      0.25
['Hydromorphone', 'Morphine', 'Heroin']      0.25
['Hydromorphone', 'Morphine', 'Buprenorphine']      0.24
['Oxymorphone', 'Morphine', 'Heroin']      0.24
['Methadone', 'Tramadol', 'Heroin']      0.24
['Oxycodone', 'Hydromorphone', 'Methadone']      0.24
['Codeine', 'Buprenorphine', 'Heroin']      0.24
['Oxycodone', 'Fentanyl', 'Methadone']      0.24
['Methadone', 'Hydrocodone', 'Hydromorphone']      0.24
['Methadone', 'Oxymorphone', 'Oxycodone']      0.24
['Codeine', 'Oxycodone', 'Morphine']      0.23
['Methadone', 'Oxymorphone', 'Hydrocodone']      0.23
['Codeine', 'Hydrocodone', 'Morphine']      0.23
['Oxycodone', 'Fentanyl', 'Tramadol']      0.23
['Methadone', 'Fentanyl', 'Hydrocodone']      0.23
['Methadone', 'Tramadol', 'Morphine']      0.23
['Methadone', 'Fentanyl', 'Heroin']      0.23
['Fentanyl', 'Heroin', 'Tramadol']      0.23
['Fentanyl', 'Hydrocodone', 'Tramadol']      0.22
['Methadone', 'Fentanyl', 'Buprenorphine']      0.22
['Fentanyl', 'Buprenorphine', 'Tramadol']      0.22
['Methadone', 'Oxymorphone', 'Buprenorphine']      0.22
['Codeine', 'Morphine', 'Heroin']      0.22
['Codeine', 'Morphine', 'Buprenorphine']      0.22
['Methadone', 'Hydromorphone', 'Heroin']      0.22
['Methadone', 'Hydromorphone', 'Morphine']      0.22
['Methadone', 'Hydromorphone', 'Buprenorphine']      0.22
['Methadone', 'Oxymorphone', 'Heroin']      0.21
['Methadone', 'Oxymorphone', 'Morphine']      0.21
['Methadone', 'Codeine', 'Oxycodone']      0.21
['Methadone', 'Codeine', 'Hydrocodone']      0.21
['Methadone', 'Fentanyl', 'Morphine']      0.21
['Codeine', 'Methadone', 'Buprenorphine']      0.20
['Methadone', 'Codeine', 'Heroin']      0.20
['Fentanyl', 'Morphine', 'Tramadol']      0.20
['Methadone', 'Codeine', 'Morphine']      0.19
['Oxycodone', 'Tramadol', 'Hydromorphone']      0.19
['Hydrocodone', 'Hydromorphone', 'Tramadol']      0.18
['Oxycodone', 'Tramadol', 'Oxymorphone']      0.18
['Oxycodone', 'Oxymorphone', 'Hydromorphone']      0.18
['Codeine', 'Tramadol', 'Oxycodone']      0.18
['Oxymorphone', 'Tramadol', 'Hydrocodone']      0.18
['Hydrocodone', 'Hydromorphone', 'Oxymorphone']      0.18
['Oxycodone', 'Fentanyl', 'Oxymorphone']      0.18
['Codeine', 'Hydrocodone', 'Tramadol']      0.18
['Codeine', 'Fentanyl', 'Oxycodone']      0.18
['Methadone', 'Fentanyl', 'Tramadol']      0.17
['Tramadol', 'Hydromorphone', 'Buprenorphine']      0.17
['Tramadol', 'Hydromorphone', 'Heroin']      0.17
['Codeine', 'Fentanyl', 'Heroin']      0.17
['Fentanyl', 'Oxymorphone', 'Hydrocodone']      0.17
['Oxymorphone', 'Buprenorphine', 'Tramadol']      0.17
['Codeine', 'Hydrocodone', 'Fentanyl']      0.17
['Oxymorphone', 'Fentanyl', 'Heroin']      0.17
['Codeine', 'Tramadol', 'Buprenorphine']      0.17
['Tramadol', 'Hydromorphone', 'Morphine']      0.17
['Oxycodone', 'Fentanyl', 'Hydromorphone']      0.17
['Oxymorphone', 'Buprenorphine', 'Fentanyl']      0.17
['Codeine', 'Tramadol', 'Heroin']      0.17
['Oxymorphone', 'Hydromorphone', 'Buprenorphine']      0.17
['Codeine', 'Fentanyl', 'Buprenorphine']      0.17
['Oxymorphone', 'Hydromorphone', 'Heroin']      0.17
['Fentanyl', 'Hydromorphone', 'Hydrocodone']      0.17
['Oxymorphone', 'Hydromorphone', 'Morphine']      0.17
['Fentanyl', 'Hydromorphone', 'Heroin']      0.17
['Tramadol', 'Oxymorphone', 'Heroin']      0.17
['Oxymorphone', 'Morphine', 'Tramadol']      0.16
['Fentanyl', 'Hydromorphone', 'Buprenorphine']      0.16
['Codeine', 'Tramadol', 'Morphine']      0.16
['Methadone', 'Tramadol', 'Hydromorphone']      0.16
['Codeine', 'Oxymorphone', 'Oxycodone']      0.16
['Codeine', 'Oxymorphone', 'Hydrocodone']      0.16
['Codeine', 'Fentanyl', 'Morphine']      0.16
['Oxymorphone', 'Morphine', 'Fentanyl']      0.16
['Codeine', 'Hydromorphone', 'Oxycodone']      0.16
['Codeine', 'Hydrocodone', 'Hydromorphone']      0.15
['Fentanyl', 'Hydromorphone', 'Morphine']      0.15
['Codeine', 'Oxymorphone', 'Buprenorphine']      0.15
['Methadone', 'Oxymorphone', 'Hydromorphone']      0.15
['Methadone', 'Tramadol', 'Oxymorphone']      0.15
['Codeine', 'Oxymorphone', 'Heroin']      0.15
['Methadone', 'Codeine', 'Tramadol']      0.15
['Codeine', 'Hydromorphone', 'Heroin']      0.15
['Codeine', 'Hydromorphone', 'Buprenorphine']      0.15
['Codeine', 'Hydromorphone', 'Morphine']      0.15
['Codeine', 'Oxymorphone', 'Morphine']      0.15
['Methadone', 'Codeine', 'Fentanyl']      0.14
['Methadone', 'Codeine', 'Hydromorphone']      0.14
['Methadone', 'Oxymorphone', 'Fentanyl']      0.14
['Methadone', 'Fentanyl', 'Hydromorphone']      0.14
['Methadone', 'Codeine', 'Oxymorphone']      0.14
['Codeine', 'Fentanyl', 'Tramadol']      0.13
['Fentanyl', 'Hydromorphone', 'Tramadol']      0.13
['Fentanyl', 'Oxymorphone', 'Tramadol']      0.13
['Oxymorphone', 'Hydromorphone', 'Tramadol']      0.12
['Codeine', 'Tramadol', 'Hydromorphone']      0.11
['Fentanyl', 'Hydromorphone', 'Oxymorphone']      0.11
['Codeine', 'Oxymorphone', 'Tramadol']      0.11
['Codeine', 'Oxymorphone', 'Hydromorphone']      0.11
['Codeine', 'Oxymorphone', 'Fentanyl']      0.11
['Codeine', 'Fentanyl', 'Hydromorphone']      0.11

Link two itemset to generate association rules. First we scan the data to statistic the quantity that both itemset appears in the same county, then we calculate the confidence of this rule using 公式confidence, if this confidence is larger than MIN_CONFIDENCE, this rule is a strong rule.
strong-rule.png
The chart above shows the distribution of confidence degree for rules.
In this paper we use MIN_CONFIDENCE=0.9 to get more credible information. And some strong rules list below.
此处适当列出部分强关联规则

['Codeine', 'Oxymorphone'] => ['Oxycodone']      conf:1.00
['Methadone', 'Hydromorphone'] => ['Oxycodone']      conf:1.00
['Methadone', 'Codeine'] => ['Oxycodone']      conf:1.00
['Oxymorphone', 'Hydromorphone'] => ['Oxycodone']      conf:1.00
['Codeine', 'Tramadol'] => ['Oxycodone']      conf:1.00
['Methadone', 'Oxymorphone'] => ['Oxycodone']      conf:1.00
['Codeine', 'Hydromorphone'] => ['Oxycodone']      conf:1.00
['Methadone', 'Tramadol'] => ['Oxycodone']      conf:1.00
['Codeine', 'Buprenorphine'] => ['Oxycodone']      conf:1.00
['Tramadol', 'Oxymorphone'] => ['Oxycodone']      conf:1.00
['Methadone', 'Morphine'] => ['Oxycodone']      conf:1.00
['Codeine', 'Oxymorphone'] => ['Hydrocodone']      conf:0.99
['Codeine', 'Fentanyl'] => ['Oxycodone']      conf:0.99
['Methadone', 'Fentanyl'] => ['Oxycodone']      conf:0.99
['Codeine', 'Morphine'] => ['Oxycodone']      conf:0.99
['Codeine', 'Hydromorphone'] => ['Hydrocodone']      conf:0.99
['Oxymorphone', 'Morphine'] => ['Oxycodone']      conf:0.99
['Tramadol', 'Hydromorphone'] => ['Oxycodone']      conf:0.99
['Oxymorphone', 'Buprenorphine'] => ['Oxycodone']      conf:0.99
['Fentanyl', 'Oxymorphone'] => ['Oxycodone']      conf:0.99
['Fentanyl', 'Hydromorphone'] => ['Oxycodone']      conf:0.99
['Methadone', 'Buprenorphine'] => ['Oxycodone']      conf:0.99
['Hydromorphone', 'Buprenorphine'] => ['Oxycodone']      conf:0.99
['Oxymorphone', 'Hydromorphone'] => ['Hydrocodone']      conf:0.99
['Oxymorphone', 'Heroin'] => ['Oxycodone']      conf:0.99
['Oxymorphone', 'Hydrocodone'] => ['Oxycodone']      conf:0.99
['Hydromorphone', 'Morphine'] => ['Oxycodone']      conf:0.99
['Tramadol', 'Morphine'] => ['Oxycodone']      conf:0.99
['Codeine', 'Morphine'] => ['Hydrocodone']      conf:0.99
['Hydrocodone', 'Hydromorphone'] => ['Oxycodone']      conf:0.99
['Hydromorphone', 'Heroin'] => ['Oxycodone']      conf:0.99
['Methadone', 'Codeine'] => ['Hydrocodone']      conf:0.99
['Codeine', 'Hydrocodone'] => ['Oxycodone']      conf:0.99
['Methadone', 'Hydrocodone'] => ['Oxycodone']      conf:0.99
['Hydrocodone', 'Tramadol'] => ['Oxycodone']      conf:0.99
['Methadone', 'Heroin'] => ['Oxycodone']      conf:0.99
['Methadone', 'Hydromorphone'] => ['Hydrocodone']      conf:0.99
['Tramadol', 'Buprenorphine'] => ['Oxycodone']      conf:0.99
['Oxymorphone'] => ['Oxycodone']      conf:0.99
['Codeine', 'Tramadol'] => ['Hydrocodone']      conf:0.99
['Methadone', 'Oxymorphone'] => ['Hydrocodone']      conf:0.99
['Morphine', 'Buprenorphine'] => ['Oxycodone']      conf:0.99
['Codeine', 'Heroin'] => ['Oxycodone']      conf:0.99
['Hydromorphone', 'Buprenorphine'] => ['Hydrocodone']      conf:0.99
['Oxymorphone', 'Tramadol'] => ['Hydrocodone']      conf:0.98
['Oxymorphone', 'Morphine'] => ['Hydrocodone']      conf:0.98
['Methadone', 'Tramadol'] => ['Hydrocodone']      conf:0.98
['Fentanyl', 'Tramadol'] => ['Oxycodone']      conf:0.98
['Tramadol', 'Hydromorphone'] => ['Hydrocodone']      conf:0.98
['Hydromorphone', 'Morphine'] => ['Hydrocodone']      conf:0.98
['Methadone', 'Morphine'] => ['Hydrocodone']      conf:0.98
['Methadone'] => ['Oxycodone']      conf:0.98
['Codeine', 'Fentanyl'] => ['Heroin']      conf:0.98
['Codeine'] => ['Oxycodone']      conf:0.98
['Codeine', 'Buprenorphine'] => ['Hydrocodone']      conf:0.98
['Morphine', 'Heroin'] => ['Oxycodone']      conf:0.98
['Hydromorphone'] => ['Oxycodone']      conf:0.98
['Hydrocodone', 'Morphine'] => ['Oxycodone']      conf:0.98
['Oxymorphone', 'Buprenorphine'] => ['Hydrocodone']      conf:0.98
['Fentanyl', 'Morphine'] => ['Oxycodone']      conf:0.98
['Fentanyl', 'Hydromorphone'] => ['Hydrocodone']      conf:0.98
['Codeine', 'Oxycodone'] => ['Hydrocodone']      conf:0.98
['Hydrocodone', 'Buprenorphine'] => ['Oxycodone']      conf:0.98
['Tramadol', 'Heroin'] => ['Oxycodone']      conf:0.98
['Methadone', 'Buprenorphine'] => ['Hydrocodone']      conf:0.98
['Tramadol', 'Morphine'] => ['Hydrocodone']      conf:0.97
['Fentanyl', 'Oxymorphone'] => ['Hydrocodone']      conf:0.97
['Oxycodone', 'Hydromorphone'] => ['Hydrocodone']      conf:0.97
['Codeine', 'Fentanyl'] => ['Hydrocodone']      conf:0.97
['Tramadol'] => ['Oxycodone']      conf:0.97
['Oxycodone', 'Oxymorphone'] => ['Hydrocodone']      conf:0.97
['Fentanyl', 'Hydrocodone'] => ['Oxycodone']      conf:0.97
['Codeine', 'Heroin'] => ['Hydrocodone']      conf:0.97
['Fentanyl', 'Hydromorphone'] => ['Heroin']      conf:0.97
['Fentanyl', 'Buprenorphine'] => ['Heroin']      conf:0.97
['Morphine'] => ['Oxycodone']      conf:0.97
['Oxymorphone', 'Heroin'] => ['Hydrocodone']      conf:0.97
['Morphine', 'Buprenorphine'] => ['Hydrocodone']      conf:0.97
['Codeine'] => ['Hydrocodone']      conf:0.97
['Fentanyl', 'Buprenorphine'] => ['Oxycodone']      conf:0.97
['Oxymorphone'] => ['Hydrocodone']      conf:0.97
['Methadone', 'Oxycodone'] => ['Hydrocodone']      conf:0.97
['Hydrocodone', 'Heroin'] => ['Oxycodone']      conf:0.97
['Hydromorphone', 'Heroin'] => ['Hydrocodone']      conf:0.97
['Fentanyl', 'Tramadol'] => ['Heroin']      conf:0.97
['Fentanyl', 'Morphine'] => ['Hydrocodone']      conf:0.97
['Buprenorphine', 'Heroin'] => ['Oxycodone']      conf:0.97
['Oxymorphone', 'Fentanyl'] => ['Heroin']      conf:0.96
['Methadone', 'Fentanyl'] => ['Hydrocodone']      conf:0.96
['Methadone', 'Heroin'] => ['Hydrocodone']      conf:0.96
['Hydromorphone'] => ['Hydrocodone']      conf:0.96
['Tramadol', 'Buprenorphine'] => ['Hydrocodone']      conf:0.96
['Codeine', 'Oxymorphone'] => ['Buprenorphine']      conf:0.96
['Methadone'] => ['Hydrocodone']      conf:0.96
['Oxycodone', 'Morphine'] => ['Hydrocodone']      conf:0.96
['Oxymorphone', 'Tramadol'] => ['Buprenorphine']      conf:0.96
['Oxycodone', 'Tramadol'] => ['Hydrocodone']      conf:0.96
['Methadone', 'Fentanyl'] => ['Heroin']      conf:0.96
['Codeine', 'Tramadol'] => ['Buprenorphine']      conf:0.96
['Oxymorphone'] => ['Oxycodone', 'Hydrocodone']      conf:0.96
['Codeine'] => ['Oxycodone', 'Hydrocodone']      conf:0.96
['Oxycodone', 'Fentanyl'] => ['Heroin']      conf:0.96
['Fentanyl', 'Morphine'] => ['Heroin']      conf:0.95
['Buprenorphine'] => ['Oxycodone']      conf:0.95
['Morphine', 'Heroin'] => ['Hydrocodone']      conf:0.95
['Hydromorphone'] => ['Oxycodone', 'Hydrocodone']      conf:0.95
['Codeine', 'Hydromorphone'] => ['Heroin']      conf:0.95
['Morphine'] => ['Hydrocodone']      conf:0.95
['Fentanyl', 'Tramadol'] => ['Hydrocodone']      conf:0.95
['Fentanyl'] => ['Heroin']      conf:0.95
['Fentanyl', 'Hydrocodone'] => ['Heroin']      conf:0.95
['Fentanyl', 'Heroin'] => ['Oxycodone']      conf:0.95
['Codeine', 'Fentanyl'] => ['Buprenorphine']      conf:0.95
['Methadone'] => ['Oxycodone', 'Hydrocodone']      conf:0.95
['Codeine', 'Oxymorphone'] => ['Heroin']      conf:0.95
['Oxymorphone', 'Fentanyl'] => ['Buprenorphine']      conf:0.95
['Codeine', 'Methadone'] => ['Buprenorphine']      conf:0.95
['Tramadol', 'Heroin'] => ['Hydrocodone']      conf:0.95
['Codeine', 'Hydromorphone'] => ['Buprenorphine']      conf:0.94
['Fentanyl'] => ['Oxycodone']      conf:0.94
['Tramadol'] => ['Hydrocodone']      conf:0.94
['Codeine', 'Tramadol'] => ['Heroin']      conf:0.94
['Methadone', 'Codeine'] => ['Heroin']      conf:0.94
['Codeine', 'Buprenorphine'] => ['Heroin']      conf:0.94
['Codeine', 'Hydromorphone'] => ['Morphine']      conf:0.94
['Fentanyl', 'Tramadol'] => ['Buprenorphine']      conf:0.94
['Codeine', 'Morphine'] => ['Heroin']      conf:0.94
['Hydrocodone'] => ['Oxycodone']      conf:0.94
['Oxymorphone', 'Hydromorphone'] => ['Buprenorphine']      conf:0.94
['Methadone', 'Fentanyl'] => ['Buprenorphine']      conf:0.94
['Fentanyl', 'Hydromorphone'] => ['Buprenorphine']      conf:0.94
['Oxycodone', 'Buprenorphine'] => ['Hydrocodone']      conf:0.94
['Methadone', 'Oxymorphone'] => ['Buprenorphine']      conf:0.94
['Oxymorphone', 'Hydromorphone'] => ['Heroin']      conf:0.93
['Methadone', 'Tramadol'] => ['Buprenorphine']      conf:0.93
['Codeine', 'Morphine'] => ['Buprenorphine']      conf:0.93
['Tramadol'] => ['Oxycodone', 'Hydrocodone']      conf:0.93
['Tramadol', 'Hydromorphone'] => ['Buprenorphine']      conf:0.93
['Tramadol', 'Hydromorphone'] => ['Heroin']      conf:0.93
['Codeine', 'Oxymorphone'] => ['Morphine']      conf:0.93
['Morphine'] => ['Oxycodone', 'Hydrocodone']      conf:0.93
['Codeine', 'Oxycodone'] => ['Heroin']      conf:0.93
['Oxymorphone', 'Hydromorphone'] => ['Morphine']      conf:0.93
['Codeine', 'Hydrocodone'] => ['Heroin']      conf:0.93
['Fentanyl', 'Buprenorphine'] => ['Hydrocodone']      conf:0.93
['Fentanyl', 'Morphine'] => ['Buprenorphine']      conf:0.93
['Oxymorphone', 'Morphine'] => ['Buprenorphine']      conf:0.92
['Codeine'] => ['Heroin']      conf:0.92
['Methadone', 'Codeine'] => ['Morphine']      conf:0.92
['Tramadol', 'Morphine'] => ['Buprenorphine']      conf:0.92
['Heroin'] => ['Oxycodone']      conf:0.92
['Oxycodone', 'Fentanyl'] => ['Hydrocodone']      conf:0.92
['Oxymorphone', 'Heroin'] => ['Buprenorphine']      conf:0.92
['Buprenorphine', 'Heroin'] => ['Hydrocodone']      conf:0.92
['Tramadol', 'Oxymorphone'] => ['Heroin']      conf:0.92
['Methadone', 'Hydromorphone'] => ['Heroin']      conf:0.92
['Hydromorphone', 'Buprenorphine'] => ['Heroin']      conf:0.92
['Codeine', 'Heroin'] => ['Buprenorphine']      conf:0.92
['Buprenorphine'] => ['Hydrocodone']      conf:0.91
['Tramadol', 'Heroin'] => ['Buprenorphine']      conf:0.91
['Codeine', 'Oxycodone'] => ['Buprenorphine']      conf:0.91
['Tramadol', 'Hydromorphone'] => ['Morphine']      conf:0.91
['Codeine'] => ['Oxycodone', 'Heroin']      conf:0.91
['Oxymorphone', 'Hydrocodone'] => ['Buprenorphine']      conf:0.91
['Codeine', 'Hydrocodone'] => ['Buprenorphine']      conf:0.91
['Oxycodone', 'Heroin'] => ['Hydrocodone']      conf:0.91
['Oxycodone', 'Oxymorphone'] => ['Buprenorphine']      conf:0.90
['Methadone', 'Hydromorphone'] => ['Morphine']      conf:0.90
['Hydromorphone', 'Morphine'] => ['Heroin']      conf:0.90
['Tramadol', 'Hydrocodone'] => ['Buprenorphine']      conf:0.90
['Methadone', 'Oxymorphone'] => ['Heroin']      conf:0.90
['Fentanyl'] => ['Oxycodone', 'Heroin']      conf:0.90
['Oxymorphone', 'Buprenorphine'] => ['Heroin']      conf:0.90
['Tramadol', 'Buprenorphine'] => ['Heroin']      conf:0.90
['Methadone', 'Hydromorphone'] => ['Buprenorphine']      conf:0.90
['Oxymorphone'] => ['Buprenorphine']      conf:0.90
['Methadone', 'Tramadol'] => ['Heroin']      conf:0.90
['Oxymorphone', 'Tramadol'] => ['Morphine']      conf:0.90
['Oxycodone', 'Tramadol'] => ['Buprenorphine']      conf:0.90
['Codeine'] => ['Buprenorphine']      conf:0.90
['Fentanyl', 'Hydromorphone'] => ['Morphine']      conf:0.90
['Codeine'] => ['Hydrocodone', 'Heroin']      conf:0.90
['Oxymorphone', 'Morphine'] => ['Heroin']      conf:0.90
['Hydromorphone', 'Morphine'] => ['Buprenorphine']      conf:0.90
['Methadone', 'Oxymorphone'] => ['Morphine']      conf:0.90
['Fentanyl'] => ['Hydrocodone']      conf:0.90
['Codeine', 'Tramadol'] => ['Morphine']      conf:0.90
['Fentanyl', 'Heroin'] => ['Hydrocodone']      conf:0.90
['Oxycodone'] => ['Hydrocodone']      conf:0.90

Predict

We use these strong rules to identify if there is any possible locations where specific opioid use might have started in each of the five states using data in 2017. The result is about 20,000, so we filtered out the substances that had already appeared in history but didn't find them in 2017. And below is our result.
predict.png
The chart above is results number with its confidence.

FIPS_Combined  already find substance  ->  predict    conference    has been appeared
54093   ['Hydrocodone', 'Buprenorphine']  ->  ['Oxycodone']  conf:0.98    [False]
21165   ['Fentanyl', 'Buprenorphine']  ->  ['Heroin']  conf:0.97    [False]
54093   ['Buprenorphine']  ->  ['Oxycodone']  conf:0.95    [False]
21165   ['Fentanyl']  ->  ['Heroin']  conf:0.95    [False]
51820   ['Codeine', 'Fentanyl']  ->  ['Buprenorphine']  conf:0.95    [False]
21171   ['Codeine', 'Fentanyl']  ->  ['Buprenorphine']  conf:0.95    [False]
54093   ['Hydrocodone']  ->  ['Oxycodone']  conf:0.94    [False]
51580   ['Hydrocodone']  ->  ['Oxycodone']  conf:0.94    [False]
51620   ['Hydrocodone']  ->  ['Oxycodone']  conf:0.94    [False]
21171   ['Codeine', 'Morphine']  ->  ['Buprenorphine']  conf:0.93    [False]
21171   ['Fentanyl', 'Morphine']  ->  ['Buprenorphine']  conf:0.93    [False]
51520   ['Tramadol', 'Oxymorphone']  ->  ['Heroin']  conf:0.92    [False]
21171   ['Codeine', 'Heroin']  ->  ['Buprenorphine']  conf:0.92    [False]
51820   ['Codeine', 'Heroin']  ->  ['Buprenorphine']  conf:0.92    [False]
51820   ['Codeine', 'Oxycodone']  ->  ['Buprenorphine']  conf:0.91    [False]
21171   ['Codeine', 'Oxycodone']  ->  ['Buprenorphine']  conf:0.91    [False]
21171   ['Codeine', 'Hydrocodone']  ->  ['Buprenorphine']  conf:0.91    [False]
21165   ['Fentanyl']  ->  ['Oxycodone', 'Heroin']  conf:0.90    [True, False]
51520   ['Oxymorphone', 'Buprenorphine']  ->  ['Heroin']  conf:0.90    [False]
21053   ['Tramadol', 'Buprenorphine']  ->  ['Heroin']  conf:0.90    [False]
51520   ['Tramadol', 'Buprenorphine']  ->  ['Heroin']  conf:0.90    [False]

Model 2

我们将各个county的统计数据与其毒品检出数量进行关联,然后使用神经网络进行拟合,得到统计数据中每个特征的权重,取前30个。通过多次神经网络的训练,我们获取出现频次最多的特征,将其按平均值分为两种类型LOW和HIGH,并将其添加到前文的模型,得到了新的关联规则。例如:

[Oxycodone HC01_VC186_LOW] -> [Hydrocodone]   conf:94
[Hydrocodone HC01_VC186_LOW] -> [Oxycodone]   conf:97.8

static {
    FAMNAMES.put(Character.valueOf('仇'), "QIU");
    FAMNAMES.put(Character.valueOf('柏'), "BO");
    FAMNAMES.put(Character.valueOf('牟'), "MU");
    FAMNAMES.put(Character.valueOf('颉'), "XIE");
    FAMNAMES.put(Character.valueOf('解'), "XIE");
    FAMNAMES.put(Character.valueOf('尉'), "YU");
    FAMNAMES.put(Character.valueOf('奇'), "JI");
    FAMNAMES.put(Character.valueOf('单'), "SHAN");
    FAMNAMES.put(Character.valueOf('谌'), "SHEN");
    FAMNAMES.put(Character.valueOf('乐'), "YUE");
    FAMNAMES.put(Character.valueOf('召'), "SHAO");
    FAMNAMES.put(Character.valueOf('朴'), "PIAO");
    FAMNAMES.put(Character.valueOf('区'), "OU");
    FAMNAMES.put(Character.valueOf('查'), "ZHA");
    FAMNAMES.put(Character.valueOf('曾'), "ZENG");
    FAMNAMES.put(Character.valueOf('缪'), "MIAO");
}

安装CUDA和CUDNN等

详见我之前的文章《Win10下安装Tensorflow 1.8 GPU版(CUDA9.2)》

捷径:

在Github上有位大神提供了windows下的编译包,可直接搜 happynear/caffe-windows ,按其教程安装即可
happynear/caffe-windows是caffe的半官方windows版,虽然在文档中只支持Visual Studio 2015, CUDA 9.0,但略作修改即可支持VS2017、CUDA9.2的环境。

不走捷径,初试Caffe

首先从官网clone源码,遵循官网教程来安装

git clone https://github.com/BVLC/caffe.git
cd caffe
git checkout windows
:: Edit any of the options inside build_win.cmd to suit your needs
C:\Projects\caffe> scripts\build_win.cmd

不出意外,肯定报错。

VisioStudio2015/Common7/Tools/../../VC/vcvarsall.bat"'
不是内部或外部命令,也不是可运行的程序 CMake Error: Error: generator : Ninja

修改scripts/build_win.cmd

  1. 第五行if DEFINED APPVEYOR是Google的自动化集成,目测用不到,跳到70行,修改为下图所示:
    CaffeScriptConfig.jpg

MSVC_VERSION=15 对应VS2017在下方会用到
WITH_NINJA=0 NINJA在Windows下不好编译安装,因此不使用
CPU_ONLY=0 如果编译CPU版本这里改为1
PYTHON_VERSION=3 目前只支持3.6,为了使用Python3.6之后还有骚操作
如果使用了Conda,从APPVEYOR部分把代码拷贝过来,修改路径;如果没有用Conda,只修改配置最后一句即可

  1. 在if %WITH_NINJA% EQU 0里添加VS2017的信息
    CaffeScriptConfig2.jpg
  2. 修改environement for VS x64
    CaffeScriptConfig3.jpg
  3. 如果使用CUDNN加速,还需要在修改cmake下的参数列表
    CaffeScriptConfig4.jpg

至此,我们再次运行scripts/build_win.cmd
CaffeScriptSuccess1.jpg
如果运气不好的话,会在这里 ERROR: Build failed,此时可以在build/目录下运行 cmake ..
然后和好运气的人一起到达下一个错误
CaffeScriptSuccess2.jpg
深入分析原因,会找到根源在 cmake/WindowsDownloadPrebuiltDependencies.cmake 因为预编译包里没有VS2017和Python3.6的属性。

编译依赖库caffe-builder

Caffe在编译时会从网上下载依赖库caffe-builder,而目前只有VS2013/2015配python2.7/3.5环境的caffe-builder。

下载最新源码

https://github.com/willyd/caffe-builder/releases

修改build_v140_x64.cmd

CaffeBuildCmd.jpg
如果编译失败,出现 CMake throws Error: could not load cache 应是缓存问题,清除缓存文件或把倒数第二行修改为 cmake ..

修改packages/protobuf/CmakeLists.txt

将protobuf包的Hash值从 14a532a7538551d5def317bfca41dace
修改为 39d6a4fa549c0cce164aa3064b1492dc
CaffeBuildConfig.jpg

修改packages/hdf5/CmakeLists.txt

将URL从 https://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8.16/src/CMkae-hdf5-1.8.16.zip
修改为 https://www.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8/hdf5-1.8.16/src/CMkae-hdf5-1.8.16.zip
CaffeBuildConfig2.jpg

运行build_v140_x64.cmd

CaffeBuildSuccess.jpg
然后进入 build_v140_x64 目录用VS2017启动 caffe-builder.sln 并生成
CaffeBuildSuccess1.jpg
此时在build/libraries目录下,生成了caffe编译所需要的文件,将这个libraries目录拷贝到C:/Users/[user name]/.caffe/dependencies下

注释掉cmake/WindowsDownloadPrebuiltDependencies.cmake中的部分内容,见下图

WindowsDownloadPrebuiltDependencies.png

重新编译Caffe,运行scripts/build_win.cmd

顺利的话,大约10分钟后就编译好了。
然而不幸的是因为有墙、引用库版本修改无法下载指定链接等原因,编译常会因为下载错误而失败。


感谢Intel教程
感谢Caffe官网

Android5.0发布后,随着Camera2的兼容性不断提升,目前完全可以替代旧API的使用。而Camera2在调用方法、相机设置、图片获取等方面都变化很大。
在实时预览图像获取时,通用的图片格式由 YUV_NV21 变为 YUV_420_888 ,因此在转换为Bitmap时出现了问题,由于Google并没有直接提供YUV2RGB的方法,因此网上代码鱼龙混杂,本文总结了几个高效的转换方法。

//Camera API
public void onPreviewFrame(byte[] data, Camera camera){
    YuvImage image = new YuvImage(data, ImageFormat.NV21, size.width, size.height, null);
}
//Camera2 API
public void onImageAvailable(ImageReader reader){
    Image image = reader.acquireLatestImage();
}

原理

HxG1S.png
cwG0i.png
可以看出NV21和YUV很相似,因此由NV21->RGB的时间与YUV->RGB的时间相似。

JAVA直接转码

该方法实现简单,处理640x480预览图平均时间为33ms

public void onImageAvailable(ImageReader reader) {
    Image image = reader.acquireLatestImage();
    if (image == null) return;
    ByteArrayOutputStream outputbytes = new ByteArrayOutputStream();

    ByteBuffer bufferY = image.getPlanes()[0].getBuffer();
    byte[] data0 = new byte[bufferY.remaining()];
    bufferY.get(data0);

    ByteBuffer bufferU = image.getPlanes()[1].getBuffer();
    byte[] data1 = new byte[bufferU.remaining()];
    bufferU.get(data1);

    ByteBuffer bufferV = image.getPlanes()[2].getBuffer();
    byte[] data2 = new byte[bufferV.remaining()];
    bufferV.get(data2);

    try {
        outputbytes.write(data0);
        outputbytes.write(data2);
        outputbytes.write(data1);
    } catch (IOException e) {
        e.printStackTrace();
    }

    final YuvImage yuvImage = new YuvImage(outputbytes.toByteArray(), ImageFormat.NV21, image.getWidth(),image.getHeight(), null);
    ByteArrayOutputStream outBitmap = new ByteArrayOutputStream();

    yuvImage.compressToJpeg(new Rect(0, 0, image.getWidth(), image.getHeight()), 95, outBitmap);
    Bitmap bitmap = BitmapFactory.decodeByteArray(outBitmap.toByteArray(), 0, outBitmap.size());
    image.close();
}

NDK方法

从理论上来说用c运算的效率会比java高,处理640x480预览图平均时间为24ms

代码详见我的Github

public void onImageAvailable(ImageReader reader) {
Image image = reader.acquireLatestImage();
Image.Plane[] plane = image.getPlanes();
byte[][] mYUVBytes = new byte[plane.length][];
for (int i = 0; i < plane.length; ++i) {
    mYUVBytes[i] = new byte[plane[i].getBuffer().capacity()];
}
int[] mRGBBytes = new int[640 * 480];

for (int i = 0; i < plane.length; ++i) {
    plane[i].getBuffer().get(mYUVBytes[i]);
}

final int yRowStride = plane[0].getRowStride();
final int uvRowStride = plane[1].getRowStride();
final int uvPixelStride = plane[1].getPixelStride();

ImageConvert.convertYUV420ToARGB8888(
        mYUVBytes[0],
        mYUVBytes[1],
        mYUVBytes[2],
        mRGBBytes,
        image.getWidth(),
        image.getHeight(),
        yRowStride,
        uvRowStride,
        uvPixelStride,
        false);

Bitmap mRGBframeBitmap = Bitmap.createBitmap(image.getWidth(), image.getHeight(), Bitmap.Config.ARGB_8888);
mRGBframeBitmap.setPixels(mRGBBytes, 0, image.getWidth(), 0, 0, image.getWidth(), image.getHeight());
image.close();
}

RenderScript方法

RenderScript、OpenGL、ndk是安卓下三大加速法宝,RenderScript运算的效率应介于java和ndk之间,处理640x480预览图平均时间为26ms

1.修改app的build文件

    defaultConfig {
        applicationId "..."
        minSdkVersion 21
        targetSdkVersion 26
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
        **renderscriptTargetApi 19**
        **renderscriptSupportModeEnabled true**
    }

2.创建RenderScript文件夹

工程列表->app->右键->New->Folder->RenderScript Folder
20170508154430300.png

3.建立rs文件 yuv420888.rs

      #pragma version(1)
      #pragma rs java_package_name(me.immi.mycare);
      #pragma rs_fp_relaxed
    
      int32_t width;
      int32_t height;
    
      uint picWidth, uvPixelStride, uvRowStride ;
      rs_allocation ypsIn,uIn,vIn;
    
     // The LaunchOptions ensure that the Kernel does not enter the padding  zone of Y, so yRowStride can be ignored WITHIN the Kernel.
     uchar4 __attribute__((kernel)) doConvert(uint32_t x, uint32_t y) {
    
     // index for accessing the uIn's and vIn's
    uint uvIndex=  uvPixelStride * (x/2) + uvRowStride*(y/2);
    
    // get the y,u,v values
    uchar yps= rsGetElementAt_uchar(ypsIn, x, y);
    uchar u= rsGetElementAt_uchar(uIn, uvIndex);
    uchar v= rsGetElementAt_uchar(vIn, uvIndex);
    
    // calc argb
    int4 argb;
        argb.r = yps + v * 1436 / 1024 - 179;
        argb.g =  yps -u * 46549 / 131072 + 44 -v * 93604 / 131072 + 91;
        argb.b = yps +u * 1814 / 1024 - 227;
        argb.a = 255;
    
    uchar4 out = convert_uchar4(clamp(argb, 0, 255));
    return out;
    }

4.rebuild项目

5.YUV_420_888_toRGB8888方法

    public static Bitmap YUV_420_888_toRGB8888(Context context, Image.Plane[] planes, byte[][] yuvBytes, int width, int height){
        
        int yRowStride= planes[0].getRowStride();
        int uvRowStride= planes[1].getRowStride();  // we know from   documentation that RowStride is the same for u and v.
        int uvPixelStride= planes[1].getPixelStride();  // we know from   documentation that PixelStride is the same for u and v.
    
        RenderScript rs = RenderScript.create(context);
        ScriptC_yuv420888 mYuv420=new ScriptC_yuv420888(rs);
    
        Type.Builder typeUcharY = new Type.Builder(rs, Element.U8(rs));
        typeUcharY.setX(yRowStride).setY(height);
        Allocation yAlloc = Allocation.createTyped(rs, typeUcharY.create());
        yAlloc.copyFrom(yuvBytes[0]);
        mYuv420.set_ypsIn(yAlloc);
    
        Type.Builder typeUcharUV = new Type.Builder(rs, Element.U8(rs));
        
        typeUcharUV.setX(yuvBytes[1].length);
        Allocation uAlloc = Allocation.createTyped(rs, typeUcharUV.create());
        uAlloc.copyFrom(yuvBytes[1]);
        mYuv420.set_uIn(uAlloc);
    
        Allocation vAlloc = Allocation.createTyped(rs, typeUcharUV.create());
        vAlloc.copyFrom(yuvBytes[2]);
        mYuv420.set_vIn(vAlloc);
    
        // handover parameters
        mYuv420.set_picWidth(width);
        mYuv420.set_uvRowStride (uvRowStride);
        mYuv420.set_uvPixelStride (uvPixelStride);
    
        Bitmap outBitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
        Allocation outAlloc = Allocation.createFromBitmap(rs, outBitmap, Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT);
    
        Script.LaunchOptions lo = new Script.LaunchOptions();
        lo.setX(0, width);  // by this we ignore the y’s padding zone, i.e. the right side of x between width and yRowStride
        lo.setY(0, height);
    
        mYuv420.forEach_doConvert(outAlloc,lo);
        outAlloc.copyTo(outBitmap);
    
        return outBitmap;
    }

6.使用示例

    public void onImageAvailable(ImageReader reader) {
        Image image = reader.acquireLatestImage();
        Image.Plane[] planes = image.getPlanes();
        for (int i = 0; i < planes.length; ++i) {
            final ByteBuffer buffer = planes[i].getBuffer();
            if (yuvBytes[i] == null) {
                yuvBytes[i] = new byte[buffer.capacity()];
            }
            buffer.get(yuvBytes[i]);
        }
        Bitmap processBitmap = UtilImage.YUV_420_888_toRGB8888(MainActivity.this,planes,yuvBytes,640,480);
    
        image.close();
    }

其他方法

还有一些其他的转换方法,效率与上面类似,不再细说

YUV->Mat->RGB

OpenGL shader实现将YUV转RGB

全球IPv4有42.9亿地址,因此随着互联网的发展,IPv4地址已不足以满足增长需要。

对于IP地址空间枯竭的问题两种延缓办法:

  1. 使用子网掩码,将网络分成更小块,提高利用率
  2. 使用NAT-> NAPT

NAT简介

NAT技术

NAT(Network Address Translation,网络地址转换)是1994年提出的。当在专用网内部的一些主机本来已经分配到了内网IP地址,但又想和因特网上的主机通信(并不需要加密)时,可使用NAT方法。 NAT的用法是有一个公网IP地址池,当有内网IP地址要向公网通讯的时候,就随机选一IP地址作为其出口IP。

更常用的NAPT技术

NAPT(Network Address Port Translation,网络端口地址转换)是一种较流行的NAT变体,是<内部地址+内部端口>与<外部地址+外部端口>之间的转换。他只有一个外部IP地址,如果有内部IP地址要出去,他就随机选一个端口来连接Internet,来表示那台内网机器。

NAT的四种类型

  1. Full Cone NAT
    内网请求(例192.168.1.2:3990)映射至路由(例60.220.123.12:45003),公网主机(例201.11.23.1:80)即可与之直接通讯(向60.220.123.12:45003发包即可);另一台公网主机(例102.1.2.3:99)向路由器映射地址(60.220.123.12:45003)发包也可以到达内网主机端口(192.168.1.2:3990)
  2. Restricted Cone NAT
    增加了安全措施,内网机器(例192.168.1.2:3990)需先向公网主机(例201.11.23.1:任意端口)发送过包,NAT路由才会转发公网主机的包,其他主机的包则丢弃;另一台公网主机(例102.1.2.3:99)向路由器映射地址(60.220.123.12:45003)发包不能到达内网主机端口(192.168.1.2:3990),必须内网机器(192.168.1.2:3990)先向公网主机(102.1.2.3:任意端口)发送过包
  3. Port-Restricted Cone NAT
    继续提升安全措施,内网机器(例192.168.1.2:3990)需先向公网主机指定端口(例201.11.23.1:80)发送过包,NAT路由才会转发公网主机80端口的包,其他端口的包则丢弃;另一台公网主机(例102.1.2.3:99)向路由器映射地址(60.220.123.12:45003)发包不能到达内网主机端口(192.168.1.2:3990),必须内网机器(192.168.1.2:3990)先向公网主机(102.1.2.3:99)发送过包
  4. Symmetric NAT
    变态提升安全性,内网主机相同端口对不同公网主机的映射端口不同,例如
    内网发包 192.168.1.2:3990->201.11.23.1:80 映射至路由 60.220.123.12:45003
    此时 60.220.123.12:45003 发包至 60.220.123.12:45003 可接受
    内网发包 192.168.1.2:3990->102.1.2.3:99 映射至路由 60.220.123.12:6033
    此时 102.1.2.3:99 发包至 60.220.123.12:6033 可接受
    但是 102.1.2.3:99 发包至 60.220.123.12:45003 被拒绝

内网穿透

NAT穿透原理

  1. 对于 Full Cone NAT
    处于不同内网的主机A和主机B,各自先连接服务器,从而在各自NAT设备上打开了一个“孔”,服务器收到主机A和主机B的连接后,知道A与B的公网地址和NAT分配给它们的端口号,然后把这些NAT地址与端口号告诉A与B;A与B即可连接对方的公网地址和端口直接进行通信。服务器在这里充当“介绍人”,告诉A与B对方的地址和端口号。
  2. 对于 Restricted Cone NAT 及 Port Restricted Cone NAT
    处于不同内网的主机A和主机B,各自先连接服务器,从而在各自NAT设备上打开了一个“孔”,服务器收到主机A和主机B的连接后,知道A与B的公网地址和NAT分配给它们的端口号,然后把这些NAT地址与端口号告诉A与B;然后主机A发送一个UDP信息到主机B的公网地址上,同时在会话期内主机B也发送一个UDP信息到主机A的公网地址上;A与B即可直接进行通信。
  3. 对于 Symmetric NAT
    处于不同内网的主机A和主机B,各自先连接服务器,服务器收到主机A和主机B的连接后,知道A与B的公网地址,然后把这些NAT地址告诉A与B;但A与B无法知道应该向哪个端口发送包,目前可用的是端口预测打孔,即会话期内主机A发送多个UDP信息到主机B的公网地址:所有端口上,同时主机B也发送多个UDP信息到主机A的公网地址:所有端口上;A与B即可直接进行通信。

常用穿透技术

UPnP协议

Universal Plug and Play,缩写为UPnP,对于一台内网电脑,UPnP功能可以使网关或路由器的NAT模块做自动端口映射,将监听的端口从网关或路由器映射到内网电脑上,同时网关或路由器的网络防火墙模块开始对Internet上其他电脑开放这个端口。

STUN协议

Simple Traversal of UDP over NATs (NAT下UDP的简单穿越)是一种网络协议,它允许位于NAT(或多重NAT)后的客户端找出自己的公网地址,查出自己位于哪种类型的NAT之后以及NAT为某一个本地端口所绑定的Internet端端口。一旦客户端得知了Internet端的UDP端口,通信就可以开始了。

UDP打洞

即使用NAT穿透原理完成内网穿透

端口转发

由服务器来转发包(消耗服务器带宽),有正向代理、反向代理等方式,类似于VPN

常用穿透软件

ngork 及各种分支

ngork官网 开源,可以自己搭建

frp

fast reverse proxy 开源,安全,可以自己搭建

花生壳

闭源,收费较贵

PP透 Peergine

商业基础服务

nat123

闭源,收费

DDNS

Dynamic Domain Name Server 是动态域名服务。DDNS定时检测用户的IP地址,然后将其与域名相对应,这样其他主机就可以通过域名来进行直接访问。
我在用的是:阿里云DDNS源码

装Ubuntu死活进不了系统,貌似是显卡问题,所以退而求其次,在Windows下装Tensorflow GPU版本。出乎意料的是这一次安装相比Tensorflow-0.8的时候容易多了,网上的教程和答疑多了很多,遇到的问题基本都能找到解决方法,同时社区对新版本的支持力度也很大。所需基础软件(对,就说的NVIDIA家)都可以放心使用最新版了!

  • 操作系统:Windows 10 64位 专业版 1803
  • Visual Studio Community 2017 版本15.7.4
  • Python:Anaconda 5.2Python 3.6
  • GPU:显卡驱动398.11-notebook-win10-64bit-international-whql + cuda_9.2.88_win10 + cudnn-9.2-windows10-x64-v7.1
  • Tensorflow: 1.8

Tips:查看Windows版本 一起按Windows键和I键,点击系统,最下方关于

一、安装vs2017

  1. CUDA指南中可以找到对VS的要求,如下图CUDA9.2目前支持Visual Studio 2017 (RTW and Update 6),因此我目前VS的版本明显高出指南的要求,但高了总比低了好。
    tensorflow-gpu-cuda-vs.png
  2. 安装VS时,按自己需求选择功能,然后要注意选择右边的Windows 10 SDK,否则CUDA的Sample会编译失败。
    tensorflow-gpu-vs.png

二、安装CUDA9.2

  1. 首先更新显卡驱动,到NVIDA官网下载显卡最新的驱动,然后安装
  2. 下载CUDA 9.2 Windows x86_64 10 exe(local),将Base Installer和 Patch 1 (Released May 16, 2018)都下载下来,得到cuda_9.2.88_win10.exe(主程序)和cuda_9.2.88.1_windows.exe(补丁包)。
  3. 右键cuda主程序,用解压软件解压到临时目录,然后运行setup.exe,选择自定义安装,只勾选CUDA下的Development、Runtime、Documentation、Samples,如下图。不选择Visual Studio Integration是因为我的VS版本高于CUDA支持版本,会有Visual Studio Integration安装失败问题,因此要绕一下,感谢知乎申尚昆的帮助,以及英伟达社区
    tensorflow-gpu-cuda.png
  4. 顺利安装完CUDA后,打开CUDA解压的临时目录,找到 CUDAVisualStudioIntegration/extras/visual_studio_integration/MSBuildExtensions 复制里面的所有文件到 VS2017 的安装目录(C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/Common7/IDE/VCVCTargets/BuildCustomizations)VS对应目录可在CUDA指南找到,如下图。
    tensorflow-gpu-vs-path.png
  5. 查看环境变量中是否有CUDA_PATH,没有的话自己加上,如下图。
    tensorflow-gpu-cuda-path.png

三、安装CUDNN

  1. 很多攻略贴中提供的CUDNN下载页面版本不全,v7.1只有Linux版,推荐从NVIDIA首页登录后找到cudnn,按网站指引下载。贴几个可能有效的地址:

官网 https://developer.nvidia.com/rdp/cudnn-download
CSDN https://download.csdn.net/download/enight0000/10490781

  1. 解压cudnn压缩包,将压缩包里的内容拷贝到CUDA的安装目录(C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v9.2)并覆盖对应的文件夹

四、测试CUDA Samples(可跳过)

  1. 打开CUDA安装目录:C:/ProgramData/NVIDIA Corporation/CUDA Samples/v9.2/5_Simulations/nbody
  2. 找到nbody_vs2017.sln,用vs2017打开,编译成功并运行,如成功则CUDA安装成功,如有错也不一定是CUDA的问题。

五、安装Tensorflow-gpu

  1. 为了Python不混乱,首先安装Anaconda 5.2(犹豫了很久怕麻烦,装上后发现真好用,一下就上手!)
  2. 设置Anaconda镜像加速下载,在CMD命令行下执行以下命令:
    conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
    conda config --set show_channel_urls yes
  3. 添加Tensorflow环境
    conda create --name tensorflow python=3.6
  4. 进入Tensorflow环境
    activate tensorflow
  5. 以下是TF的安装步骤,如已安装VS2017推荐安装TF1.8(无坑),否则就安装最新的TF1.12(有小坑),请酌情选一个吧~

TF1.8安装步骤)安装Tensorflow-gpu1.8,由于TF官网的GPU版本只支持到VS2015,而我们想使用VS2017,因此使用别人在CUDA9.2下编译好的安装包

https://github.com/fo40225/tensorflow-windows-wheel/tree/master/1.8.0/py36/GPU/cuda92cudnn71sse2

下载后输入 pip install xxx.whl 安装

TF1.12安装步骤)Tensorflow1.12官网对windows的支持已经很好了,可直接使用 pip install tensorflow-gpu 这条命令安装,但需要预装Microsoft Visual C++ 2015 Redistributable 更新 3,以下是官网说明。

安装 Microsoft Visual C++ 2015 Redistributable 更新 3。此软件包随附在 Visual Studio 2015 中,但可以单独安装:
转到 Visual Studio 下载页面 链接
选择“可再发行组件和生成工具”
下载并安装 Microsoft Visual C++ 2015 Redistributable 更新 3。

附:可能遇到的问题

问题一:从1.8升级至1.12后或直接安装1.12后,运行 import tensorflow 报错

from tensorflow.python.pywrap_tensorflow_internal import *
...
ImportError: DLL load failed: 找不到指定的模块

是因为VS版本不匹配造成的,请注意是否为VS2015 Update3,以及步骤二中的4小节

问题二:在使用PowerShell时 activate tensorflow 输入后没有效果,而在 anaconda prompt 和 cmd 中均是可以的

解决方案:

  1. 安装包

    conda install -n root -c pscondaenvs pscondaenvs

  2. 更改PowerShell配置
    以管理员身份启动PowerShell,并输入命令个 Set-ExecutionPolicy RemoteSigned

执行策略更改选择Y
然后就可以使用activate了

六、测试

tensorflow-gpu-tf.png

网上安装教程鱼龙混杂,多数都有莫名其妙的坑无法走通,因此记录一下配置过程。
Jupyter Notebook仅可同时一人使用,多人使用会同时修改一个文件;多人使用请关注JupyterHub
自建外网访问需要路由支持公网IP+映射端口,否则请使用Holer直接转发

已经过测试的系统有:
CentOS7 64位
Redhat Enterprise Linux6 64位
如果这篇文章能帮您成功完成安装,请您留言告知您的系统环境~
如遇到问题更欢迎留言~


在阿里云购买ECS服务器不多说,设置系统密码,并登陆进系统

第一步:安装SSH(可跳过)

阿里自带的浏览器控制延迟很大,体验极差
我选择使用SSH并用Putty登陆

CentOS中安装并启动SSH

rpm -qa |grep ssh   #查看SSH是否安装
yum install openssh-server  #安装SSH
service sshd start  #启动SSH服务,stop为停止
netstat -antp |grep sshd  #查看是否启动22端口,会显示一条在22端口监听的信息
iptables -nL  #看网络策略是否禁止了22端口,不显示为成功
chkconfig sshd on #设置SSH服务为开机启动

下载Putty并连接CentOS

https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

连接并测试

为了安全起见,修改SSH服务端口

vim /etc/ssh/sshd_config 

找到“#Port 22”修改为
Port 22
Port 20000
注意去掉#
即保留22端口,同时增加20000端口

重启服务

service sshd restart

修改安全策略

在阿里云-云服务器 ECS-安全组-配置规则-快速创建规则
自定义端口 TCP 20000
授权 地址段访问 0.0.0.0/0

连接并测试

若成功,则删除安全策略中22端口的配置,删除sshd_config中的22端口

第二步:安装Python3(亦可以安装miniconda)

查看系统信息

查看自带Python版本,得到Python2.7.5

python

查看Python安装路径,得到位于/usr/bin/python目录下

which python

由于centos7原本就安装了Python2,且有很多系统命令使用Python2,因此不删除Python2。
同时为了使用Tensorflow,Jupyter服务基于Python3安装。

安装基础依赖环境

yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel

下载

浏览器打开 https://www.python.org/ftp/python/ 查看寻找Python版本,由于Tensorflow还不支持Python3.6版本,因此安装Python3.5.4

wget https://www.python.org/ftp/python/3.5.4/Python-3.5.4.tgz

创建Python3的目录

mkdir /usr/local/python3 

解压下载文件并进入解压后目录

tar -zxvf Python-3.5.4.tgz
cd Python-3.5.4

执行自定义安装

./configure --prefix=/usr/local/python3
make && make install

创建软链接

ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3

测试命令

python3
pip3

第三步:安装Jupyter

更新pip

pip3 install --upgrade pip

安装Jupyter同时会安装IPython和notebook

pip3 install jupyter

查看python3的安装路径,应该在/usr/local/python3

cd /usr/local #跳转目录
ls #查看文件列表

创建软连接,否则会出现juptyer command not found

ln -sv /usr/local/python3/bin/ipython /usr/bin/
ln -sv /usr/local/python3/bin/jupyter /usr/bin/

创建Jupyter配置文件,生成在/root/.jupyter/jupyter_notebook_config.py

jupyter notebook --generate-config

打开配置文件

cd /root/.jupyter/
vim jupyter_notebook_config.py

进行如下修改

注意去掉前面的#
c.NotebookApp.port = 20006
c.NotebookApp.ip = '你的服务器公网IP' #或填"*"允许ip访问
c.NotebookApp.open_browser = False
修改Jupyter文件默认存储路径
c.NotebookApp.notebook_dir = '/opt/'

vim操作教程
输入i进入编辑模式,按esc键退出编辑模式
输入:/xxx 查找xxx
输入:wq 保存退出

(非必须)使用Jupyter打印日志较多时会报 IOPub data rate exceeded 错误,因为日志打印太快可能导致客户端崩溃,但很不方便,因此做一下修改:
iopub_data_rate_limit = 100000000 #去掉注释并且将后面的限制值多加几个0

(非必须)当使用conda作为包管理器时,需要notebook支持在多个不同环境间切换。输入命令安装nb_conda,之后启动notebook即可看到kernels的切换功能。

conda install nb_conda_kernels

设置登录密码

jupyter notebook password

后台启动 notebook

(nohup jupyter notebook --allow-root --ip=0.0.0.0 > deep.log &)
按两次回车即可
#"nohup", 即 “nohup ./myjob &”,忽略hangup信号,防止shell关闭时程序停掉

查看Jupyter日志

cat deep.log

修改安全策略

在阿里云-云服务器 ECS-安全组-配置规则-快速创建规则
自定义端口 TCP 20006
授权 地址段访问 0.0.0.0/0

连接并测试

在浏览器输入 公网ip:端口

设置

pd.set_option('display.mpl_style', 'default') # Make the graphs a bit prettier
#option display.mpl_style providing a sleeker visual style for plots, based on [GH3075][1] 
pd.set_option('display.line_width', 5000) 
pd.set_option('display.max_columns', 60) 

DataFrame

df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=['A','B','C','D'])
df = pd.read_csv(path_train, encoding="utf-8", header=0)
df.to_csv(path,columns=['Pred'], index=True, header=True)
df.info()#类型
df.describe()#统计

时间转换

data['TIME'] = pd.to_datetime(data['TIME'], unit='s') #将Unix毫秒数据转为datetime格式
data['TIME'].astype('<M8[ns]') #转换列类型
data['HOUR'] = [i.hour for i in data["TIME"]] #从datetime中取小时

重命名列名

df.index.name = 'Id' #重命名索引列
df.columns = ['Id', 'Pred'] #重命名所有列
df.rename(columns={'Y':'Pred'}, inplace = True) #重命名指定列

选取DataFrame信息

#取列
df['A']#得到Series类型
df[['A','B']]#得到DataFrame类型
#取行
df.loc[100],df.ix[100]  #按index值取Series
df.iloc[0]          #按位置选Series
df.loc[[100,101,102]],df.loc[100:103]
df.iloc[[0,1,2]],df.iloc[0:3]
#取值
df.loc[100]['A'],df.loc[100,'A']#float64
df.loc[100,['A']],df.ix[100:102,['A','B']]#Series/DataFrame
df.iloc[0]['A']#取值
df.iloc[0:2][['A','B']]#取值

破解步骤:

  1. 用jadx打开apk发现在assetsbinDataManagedUnityEngine.dll这个文件,推测是一款unity游戏
  2. 用Reflector逆向Assembly-CSharp.dll发现代码都是明文的
  3. 修改代码实现想要的功能就好啦,然后用新的dll文件重新编译apk,签名就ok了

魔改版介绍:

  1. 游戏名修改为《你家青蛙》
  2. 每次打开游戏都是新的一天
  3. 每次打开游戏+500三叶草
  4. 抽奖仅消耗一张奖券
  5. 邮箱、抽奖获得奖券的数量x5
  6. 更多部分的汉化
  7. 与原版可并存

地址:

本次破解仅为学习之用,娱乐就好~
魔改会无聊,开挂需谨慎~
https://pan.lanzou.com/i0pvokf

反复开几次游戏,就能玩一个月的进度....
微信图片_20180325223725.jpg

游戏脚本制作的软件不是很多,大部分都要收费,所以我就用的按键精灵9,因为它封装了不少图像匹配的功能,可以从界面来进行自动化操作。
temp.jpg
下面附上原理:
1.界面识别需要标志物,这里我是用鱼竿顶端的绿灯作为标志(游戏中是使用技能点亮的)【误差:绿灯颜色会随位置、光线、角度的不同而有偏差,例:某地区不同光线颜色范围由20FFFF~5EFFFF】
2.当标志位置几乎不变的时候判断为没有鱼上钩,当标志位置移动较大时判断为有鱼咬饵【误差:鱼竿会随风摇摆,不同视角看到的偏移是不同的】
3.收杆动作有多种,耗时不同,所以多次延时尝试抛饵【误差:脱钩4s收回,普通鱼8s收回,优品鱼可能到11s】

代码结构如下:
1.寻找游戏句柄
2.在游戏窗口中寻找鱼竿顶端绿灯
3.等待玩家收杆,并开始自动钓鱼
For 次数{

    4.抛竿
    5.寻找绿灯标志
    6.找到标志后,进入标志位移判断阶段
    7.标志位移超过常数,表明有鱼咬钩,拉杆
    8.延时等待

功能扩展:
1.由于绿灯颜色根据游戏中光线明亮会有变化,增加颜色偏差指数
2.由于可能存在步骤5的无限循环,所以加入第二个线程。在进入步骤5时开启线程2,若5秒后尝试再次抛饵,10秒后报警提示

可扩展功能:
步骤2中加入标志位移测算
鱼饵用尽后自动更换
钓鱼一定时间后自动位移,防鱼洞枯竭

源码:

UserVar precision=0.9 "颜色匹配精度0.75-1"
UserVar range = 13 "鱼鳔浮动范围10-30"
UserVar times = 100 "钓鱼次数"
Global watch
DimEnv GameTable
GameTable = 0
dim MyArray
Dim x
Dim y
x = - 1 
y = - 1 
Dim XX
Dim YY
XX = - 1 
YY = - 1 
Dim z
z = 0

//进程查找
Call Plugin.Msg.Tips("查找游戏进程中")
While GameTable = 0
GameTable = Plugin.Window.Find(0, "最终幻想XIV")
Delay 500
Wend

Call Plugin.Msg.Tips("进城找到,句柄=" & GameTable)

20141201144557703.jpg