pathtree: Make dataitems() use the cache dict under restricted args

This achieves a 100x improvement in iteration time when the subtree
rooted at the root of the full tree and there is no depth limi

pathtree: Make dataitems() use the cache dict under restricted args

This achieves a 100x improvement in iteration time when the subtree
rooted at the root of the full tree and there is no depth limit (i.e.
the caller has requested all "populated" items). Whilst it sounds like a
restricted case, it is a very common query by way of phosphor-objmgr's
obmc/mapper/server.py:ObjectMapper.process_old_owner()

Note this is a quasi API break, as some keys that were previously
not present in the results may now appear: Explicitly storing 'None'
into the data structure will have the key with 'None' appear in the
full-tree dataitems() case. However, given this was filtered previously,
no existing callers should be storing 'None' into the tree as they would
not have been able to retrieve it via dataitems().

Before:
$ python -m obmc.utils.testpathtree
...
Iteration tests:
depth=1, width=1, n=1000: 0.135853052139
depth=1, width=2, n=1000: 0.203811883926
depth=1, width=3, n=1000: 0.26814198494
depth=1, width=4, n=1000: 0.333888053894
depth=2, width=1, n=1000: 0.193987131119
depth=2, width=2, n=1000: 0.264018058777
depth=2, width=3, n=1000: 0.327262878418
depth=2, width=4, n=1000: 0.38805603981
depth=3, width=1, n=1000: 0.253651857376
depth=3, width=2, n=1000: 0.317117929459
depth=3, width=3, n=1000: 0.385557889938
depth=3, width=4, n=1000: 0.452265024185
depth=4, width=1, n=1000: 0.327889919281
depth=4, width=2, n=1000: 0.390358924866
depth=4, width=3, n=1000: 0.459683895111
depth=4, width=4, n=1000: 0.530153989792

After:
$ python -m obmc.utils.testpathtree
...
Iteration tests:
depth=1, width=1, n=1000: 0.0012412071228
depth=1, width=2, n=1000: 0.00455403327942
depth=1, width=3, n=1000: 0.00307989120483
depth=1, width=4, n=1000: 0.00356507301331
depth=2, width=1, n=1000: 0.00118088722229
depth=2, width=2, n=1000: 0.00169396400452
depth=2, width=3, n=1000: 0.00234699249268
depth=2, width=4, n=1000: 0.00300288200378
depth=3, width=1, n=1000: 0.00100708007812
depth=3, width=2, n=1000: 0.00161695480347
depth=3, width=3, n=1000: 0.00234794616699
depth=3, width=4, n=1000: 0.00315403938293
depth=4, width=1, n=1000: 0.00101804733276
depth=4, width=2, n=1000: 0.00204801559448
depth=4, width=3, n=1000: 0.00281095504761
depth=4, width=4, n=1000: 0.0070219039917

Change-Id: Ice3afd12e2b112227735f0f1dedb6a8ea594740c
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

show more ...

pathtree: Allow get() on subtrees

When running recent PathTree modifications on a BMC we can see failures
in the log such as "Failed to invoke ObjectMapper method" from the
phosphor-fru-fault-monito

pathtree: Allow get() on subtrees

When running recent PathTree modifications on a BMC we can see failures
in the log such as "Failed to invoke ObjectMapper method" from the
phosphor-fru-fault-monitor. This can be triggered via the
openbmc-test-automation suite, for example with the following command

$ robot -v OPENBMC_HOST:w5.aus.stglabs.ibm.com --include \
Create_Test_Error_Callout_And_Verify extended/test_association.robot

Capturing the DBus traffic across this test case
(`busctl capture > dbus.pcap`) and analysing it with Wireshark showed the
following method invocation:

```
0000 6c 01 00 01 53 00 00 00 16 00 00 00 c6 00 00 00 l...S...........
0010 01 01 6f 00 22 00 00 00 2f 78 79 7a 2f 6f 70 65 ..o.".../xyz/ope
0020 6e 62 6d 63 5f 70 72 6f 6a 65 63 74 2f 6f 62 6a nbmc_project/obj
0030 65 63 74 5f 6d 61 70 70 65 72 00 00 00 00 00 00 ect_mapper......
0040 03 01 73 00 09 00 00 00 47 65 74 4f 62 6a 65 63 ..s.....GetObjec
0050 74 00 00 00 00 00 00 00 02 01 73 00 20 00 00 00 t.........s. ...
0060 78 79 7a 2e 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a xyz.openbmc_proj
0070 65 63 74 2e 4f 62 6a 65 63 74 4d 61 70 70 65 72 ect.ObjectMapper
0080 00 00 00 00 00 00 00 00 06 01 73 00 20 00 00 00 ..........s. ...
0090 78 79 7a 2e 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a xyz.openbmc_proj
00a0 65 63 74 2e 4f 62 6a 65 63 74 4d 61 70 70 65 72 ect.ObjectMapper
00b0 00 00 00 00 00 00 00 00 08 01 67 00 03 73 61 73 ..........g..sas
00c0 00 00 00 00 00 00 00 00 07 01 73 00 05 00 00 00 ..........s.....
00d0 3a 31 2e 35 37 00 00 00 20 00 00 00 2f 78 79 7a :1.57... .../xyz
00e0 2f 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a 65 63 74 /openbmc_project
00f0 2f 6c 65 64 2f 67 72 6f 75 70 73 2f 00 00 00 00 /led/groups/....
0100 27 00 00 00 22 00 00 00 6f 72 67 2e 66 72 65 65 '..."...org.free
0110 64 65 73 6b 74 6f 70 2e 44 42 75 73 2e 4f 62 6a desktop.DBus.Obj
0120 65 63 74 4d 61 6e 61 67 65 72 00 ectManager.
```

and the response

```
0000 6c 03 01 01 3f 00 00 00 9d 08 00 00 5e 00 00 00 l...?.......^...
0010 06 01 73 00 05 00 00 00 3a 31 2e 35 37 00 00 00 ..s.....:1.57...
0020 04 01 73 00 27 00 00 00 6f 72 67 2e 66 72 65 65 ..s.'...org.free
0030 64 65 73 6b 74 6f 70 2e 44 42 75 73 2e 45 72 72 desktop.DBus.Err
0040 6f 72 2e 46 69 6c 65 4e 6f 74 46 6f 75 6e 64 00 or.FileNotFound.
0050 05 01 75 00 16 00 00 00 08 01 67 00 01 73 00 00 ..u.......g..s..
0060 07 01 73 00 05 00 00 00 3a 31 2e 31 35 00 00 00 ..s.....:1.15...
0070 3a 00 00 00 70 61 74 68 20 6f 72 20 6f 62 6a 65 :...path or obje
0080 63 74 20 6e 6f 74 20 66 6f 75 6e 64 3a 20 2f 78 ct not found: /x
0090 79 7a 2f 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a 65 yz/openbmc_proje
00a0 63 74 2f 6c 65 64 2f 67 72 6f 75 70 73 2f 00 ct/led/groups/.
```

These packets translate to the following `busctl` invocation and response:

```
root@witherspoon:~# busctl call \
xyz.openbmc_project.ObjectMapper \
/xyz/openbmc_project/object_mapper \
xyz.openbmc_project.ObjectMapper \
GetObject sas \
/xyz/openbmc_project/led/groups/ \
1 \
org.freedesktop.Dbus.ObjectManager
path or object not found: /xyz/openbmc_project/led/groups/
root@witherspoon:~#
```

From `busctl tree` we can see that the objects do actually appear on the
bus:

root@witherspoon:~# busctl tree
...
Service xyz.openbmc_project.LED.GroupManager:
└─/xyz
└─/xyz/openbmc_project
└─/xyz/openbmc_project/led
└─/xyz/openbmc_project/led/groups
├─/xyz/openbmc_project/led/groups/bmc_booted
├─/xyz/openbmc_project/led/groups/bmc_fault
├─/xyz/openbmc_project/led/groups/boxelder_fault
├─/xyz/openbmc_project/led/groups/core0_fault
├─/xyz/openbmc_project/led/groups/core10_fault
...
```

However, the /xyz/openbmc_project/led/groups/ path represents a subtree
root, not a leaf element. I didn't realise this was a supported
use-case, and over-aggressively removed support for it in the recent
patch adding the flat dict cache.

Add in a fallback to iterating the PathTree datastructure when we take a
cache miss.

Change-Id: I658c3fdd6b87be9d814c59a654e63ec5bb83e2bd
Fixes: 52aeb314c0eb ("pathtree: Cache keys in a flat dict for fast lookup")
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

show more ...

testpathtree: Add dataitems() stress test

Change-Id: I7493a8d2d9df6123a3d80f17b2ba268608997066
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

pathtree: Cache keys in a flat dict for fast lookup

This knocks about a second off of the depth stress tests:

$ python -m obmc.utils.testpathtree
Depth tests:
1: 5.18097305298
2: 6.

pathtree: Cache keys in a flat dict for fast lookup

This knocks about a second off of the depth stress tests:

$ python -m obmc.utils.testpathtree
Depth tests:
1: 5.18097305298
2: 6.18765211105
3: 7.3427259922
4: 8.33903598785
5: 9.55236506462
6: 10.7165560722
7: 12.1563079357
8: 13.3570129871
9: 14.7827298641
10: 16.0967490673

Width tests:
1: 1.20649981499
2: 1.20889282227
4: 1.20902085304
8: 1.22063994408
16: 1.22446990013
32: 1.21649098396
64: 1.21024799347
128: 1.21360588074
256: 1.22187900543
512: 1.23233604431
1024: 1.21607708931
2048: 1.21069407463
4096: 1.22389698029
8192: 1.20828580856
16384: 1.21290493011
32768: 1.21552395821
65536: 1.23201704025
131072: 1.21459794044
262144: 1.24190402031
524288: 1.21342992783
1048576: 1.21248602867

However, we get just over a 300% improvement in the host reboot
scenario, which is captured by some separate tools used to analyse the
PathTree's performance issues:

$ # Before
$ ./replay.py -n 1000 ptinit.log ptreboot.log
deep copy: 1000 iterations took 3.0744523939938517s
ops script: 1000 iterations took 10.03650910500437s
net: 1000 iterations took 6.9620567110105185s
$ # After
$ ./replay.py -n 1000 ptinit.log ptreboot.log
deep copy: 1000 iterations took 6.784729664999759s
ops script: 1000 iterations took 9.0453162470003s
net: 1000 iterations took 2.260586582000542s

Change-Id: I6826b99950fa1561d292908c4ffff1137ecaa1bc
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

show more ...

testpathtree: Add stress tests

Indicative performance on a lightly loaded 3rd Gen Lenovo X1 Carbon:

$ # Timing 1,000,000 executions of the test case, time units are seconds
$ python -m obmc.utils.t

testpathtree: Add stress tests

Indicative performance on a lightly loaded 3rd Gen Lenovo X1 Carbon:

$ # Timing 1,000,000 executions of the test case, time units are seconds
$ python -m obmc.utils.testpathtree
Depth tests:
1: 6.01530885696
2: 7.11315083504
3: 8.41325497627
4: 9.51180887222
5: 10.7866010666
6: 12.0375499725
7: 13.4880149364
8: 14.7702541351
9: 16.3227319717
10: 17.6581590176

Width tests:
1: 1.19656515121
2: 1.20851492882
4: 1.2031879425
8: 1.19995284081
16: 1.21298193932
32: 1.20965600014
64: 1.20766687393
128: 1.21984100342
256: 1.21412611008
512: 1.21589684486
1024: 1.2073469162
2048: 1.2244079113
4096: 1.21275901794
8192: 1.22486519814
16384: 1.21715903282
32768: 1.21435189247
65536: 1.2201769352
131072: 1.21944999695
262144: 1.21434497833
524288: 1.20947313309
1048576: 1.21527695656

Change-Id: I05aa3fb59ddc2510e22e22a8802e166e2e801bab
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

show more ...

utils: Add characterisation tests for PathTree

Change-Id: I1ad4832779e85a54a75b3551a8beeba429679e1c
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

utils: Remove ListMatch

This is a pointless class given python lambdas. It was also
broken with aea38c65. Drop.

Change-Id: Ib269fee1b09cd04defe6ec98c2f5dc4118d5425e
Signed-off-by: Brad Bishop <br

utils: Remove ListMatch

This is a pointless class given python lambdas. It was also
broken with aea38c65. Drop.

Change-Id: Ib269fee1b09cd04defe6ec98c2f5dc4118d5425e
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

Enable pycodestyle

Apply fix-ups and run pycodestyle during CI testing.

Change-Id: I1005495b11e228abdc8d40a51dbf81c4a6e6c92c
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

Port to python 3

Port code to python 3 yet remain backward compatible to python 2.
This port is in preparation for yocto 2.4 upgrade.

Resolves openbmc/openbmc#2920

Change-Id: Ie06d206a3de3de2ea636

Port to python 3

Port code to python 3 yet remain backward compatible to python 2.
This port is in preparation for yocto 2.4 upgrade.

Resolves openbmc/openbmc#2920

Change-Id: Ie06d206a3de3de2ea636b043c099a0185b32ce53
Signed-off-by: CamVan Nguyen <ctnguyen@us.ibm.com>

show more ...

Revert "Port obmc to python3.5"

Parse error with this change:
TypeError: iter() returned non-iterator of type 'PathTreeItemIterator'

This reverts commit f99783be6592130bbad2d0f69ada2b7ac61618f3.

C

Revert "Port obmc to python3.5"

Parse error with this change:
TypeError: iter() returned non-iterator of type 'PathTreeItemIterator'

This reverts commit f99783be6592130bbad2d0f69ada2b7ac61618f3.

Change-Id: Id2b7f4af5284d25d59e19905c7b873726b02f700
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

Port obmc to python3.5

Change-Id: I1ed6c6bd914635485a697369048a8be9e5919fc6
Signed-off-by: Balaji B Rao <balajibapu@gmail.com>

utils: add prefix

Allow some composition of the strings returned by
org_dot_openbmc_match_strings.

Signed-off-by: Brad Bishop <bradleyb
Change-Id: I8e6a522af175bba70aedbfb4f5a65e06b9237598

utils: match parents

Match parents like '/org' in addition to '/org/openbmc/foo' when
testing with org_dot_openbmc_match.

Change-Id: I707f1de0367bfad0de4afa4436b10203c0d7f852
Signed-off-by: Brad Bi

utils: match parents

Match parents like '/org' in addition to '/org/openbmc/foo' when
testing with org_dot_openbmc_match.

Change-Id: I707f1de0367bfad0de4afa4436b10203c0d7f852
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

utils: provide dbus match strings

Provide a list of name filters, suitable for use with dbus
interface/path name matching.

Change-Id: I47a3be43e4460e35ec457436300fac82eb18a03f
Signed-off-by: Brad B

utils: provide dbus match strings

Provide a list of name filters, suitable for use with dbus
interface/path name matching.

Change-Id: I47a3be43e4460e35ec457436300fac82eb18a03f
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

Remove Path class

Unused, and os.path should be favored anyway.

Change-Id: I9ba718e15695d9e14b8893ef09721fb4f90c2b85
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

Add xyz.openbmc-project to interface match

In preparation for migration to xyz.openbmc-project.

Change-Id: Ia134de704affb3b51726790ea38b9b36bdda16b1
Signed-off-by: Brad Bishop <bradleyb@fuzziesquir

Add xyz.openbmc-project to interface match

In preparation for migration to xyz.openbmc-project.

Change-Id: Ia134de704affb3b51726790ea38b9b36bdda16b1
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

Added python to dts encoder

This is a rudimentary python to device tree encoder.

It supports nested nodes and, cell, cell array,
string, string array, and mixed array properties.
At the moment ther

Added python to dts encoder

This is a rudimentary python to device tree encoder.

It supports nested nodes and, cell, cell array,
string, string array, and mixed array properties.
At the moment there is no support for binary properties.

show more ...

Add dictionary export to PathTree

PathTree is a dictionary extension with paths as keys and lets
you do things like iterate on subpaths, etc. Internally it is
implemented as a nested dictionary but

Add dictionary export to PathTree

PathTree is a dictionary extension with paths as keys and lets
you do things like iterate on subpaths, etc. Internally it is
implemented as a nested dictionary but on the outside it acts
like a normal dictionary.

This change enables exporting a PathTree structure as a nested
dictionary, which is useful for python_2_xyz encoders that understand
nested data structures.

show more ...

Introducing pyobmc

We have openbmc python utilities scattered all over the place. To
facilitate reuse, bring them together in a single python package.

None of this is new code, it was all simply p

Introducing pyobmc

We have openbmc python utilities scattered all over the place. To
facilitate reuse, bring them together in a single python package.

None of this is new code, it was all simply ported and re-arranged
from other projects.

Ran everything through pep8.

show more ...